Connecting unmanaged segmented networks by managing interconnection switching elements

ABSTRACT

Some embodiments provide a novel network control system for interconnecting several separate networks. The system includes a set of interconnection switching elements. Each interconnection switching element in the set is for connecting one of the separate networks to a common interconnecting network. The system includes a set of network controllers for managing the interconnection switching elements in order for the interconnection switching elements to send packets from a first machine at a first one of the networks to a second machine at a second one of the networks.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application claims the benefit of U.S. Provisional Application61/524,755, entitled “Interconnecting Logical Datapaths at MultipleFailure Domains”, filed Aug. 17, 2011; U.S. Provisional Application61/524,756, entitled “Virtual Interconnect for Binding TogetherDifferently Segmented Site Networks”, filed Aug. 17, 2011; U.S.Provisional Application 61/671,664, entitled “Hierarchical LogicalSwitching Structure”, filed Jul. 13, 2012; and U.S. ProvisionalApplication 61/675,807, entitled “Interconnection of Networks”, filedJul. 25, 2012. U.S. Applications 61/524,755, 61/524,756, 61/671,664, and61/675,807 are incorporated herein by reference.

BACKGROUND

Many current enterprises have large and sophisticated networkscomprising switches, hubs, routers, servers, workstations and othernetworked devices, which support a variety of connections, applicationsand systems. The increased sophistication of computer networking,including virtual machine migration, dynamic workloads, multi-tenancy,and customer specific quality of service and security configurationsrequire a better paradigm for network control. Networks havetraditionally been managed through low-level configuration of individualcomponents. Network configurations often depend on the underlyingnetwork: for example, blocking a user's access with an access controllist (“ACL”) entry requires knowing the user's current IP address. Morecomplicated tasks require more extensive network knowledge: forcingguest users' port 80 traffic to traverse an HTTP proxy requires knowingthe current network topology and the location of each guest. Thisprocess is of increased difficulty where the network switching elementsare shared across multiple users.

In response, there is a growing movement towards a new network controlparadigm called Software-Defined Networking (SDN). In the SDN paradigm,a network controller, running on one or more servers in a network,controls, maintains, and implements control logic that governs theforwarding behavior of shared network switching elements on a per userbasis. Making network management decisions often requires knowledge ofthe network state. To facilitate management decision-making, the networkcontroller creates and maintains a view of the network state andprovides an application programming interface upon which managementapplications may access a view of the network state.

Some of the primary goals of maintaining large networks (including bothdatacenters and enterprise networks) are scalability, mobility, andmulti-tenancy. Many approaches taken to address one of these goalsresults in hampering at least one of the others. For instance, one caneasily provide network mobility for virtual machines within a layer 2(“L2”) domain, but L2 domains cannot scale to large sizes. Furthermore,retaining tenant isolation greatly complicates mobility. As such,improved solutions that can satisfy the scalability, mobility, andmulti-tenancy goals are needed.

BRIEF SUMMARY

Some embodiments of the invention provide different mechanisms forconnecting two or more networks together in order to achievecommunication between machines (e.g., virtual machines or physicalmachines) located within the different networks. In some embodiments,the networks connected together are located at different sites (e.g.,different data centers), each of which consists of several end machines.Some embodiments utilize interconnection switching elements located atthe edges of the different networks that are specifically programmed toenable packets from a source network to be read by a destinationnetwork. In some embodiments, the connection of the networks forms alarger logical network that includes the individual site networks.

In some embodiments, the individual networks logically connectedtogether are themselves logical networks. Each logical network ismanaged by a network control system that enables the specification of alogical datapath set for the logical network. The network control systemconfigures a set of shared switching elements to implement the specifiedlogical datapath set, thereby virtualizing the switching elements. Inorder to interconnect multiple logical networks, some embodimentsprovide a hierarchical control system that includes a hierarchicalarrangement of network controllers. A set of higher-level networkcontrollers receives a specification of a higher-level logical datapathset that includes the machines of all of the lower-level logicalnetworks. The higher-level network controllers generate flow entriesthat are passed down to the lower-level network controllers. Theselower-level network controllers configure the shared switching elementswithin their respective networks in order to implement the higher-levellogical datapath set on top of their own respective lower-level logicaldatapath sets. In this case, the higher-level logical datapath setserves to interconnect the lower-level logical datapath sets, therebyinterconnecting the networks.

The hierarchical controller arrangement in turn results in hierarchicalprocessing of packets by the switching elements within the network. Asstated, the higher-level network controllers generate flow tables thatimplement the highest-level logical datapath set that governs packetforwarding between end machines of the network, then pass these tablesdown to the lower-level network controllers. The lower-level networkcontrollers then incorporate the received higher-level flow entries intotheir own flow tables that implement the lower-level logical datapathsets. Each set of lowest-level network controllers generates flow tablesfor the physical managed switching elements within the network of thecontroller set, and passes these generated flow tables to the physicalswitching elements. These flow tables specify lookup entries that themanaged switching elements use to process packets within the logicalnetwork. When a managed switching element receives a packet on aphysical port, the managed switching element maps the packet to logicalports of the hierarchical data paths, makes a forwarding decision to anegress port at the highest level, and maps the identified egress portback down through the hierarchy of data paths to a physical egress port.

In some embodiments, the networks logically connected together mayinclude unmanaged segmented networks. In some cases, a network managerwill want to provide connections between a first network segmented usinga first tagging or tunneling technique (e.g., VLAN, MAC-in-MAC, L2 overL3, MPLS, etc.) and a second network segmented using a second tagging ortunneling technique. Even when the two networks use the same technique(e.g., both networks using VLANs), the implementation of that technique(e.g., the structure of the tags used in packet headers) may bedifferent between the two networks such that they are effectively usingtwo different techniques. Some embodiments provide a mechanism forconnecting such differently-segmented networks across a commoninterconnecting network (e.g., a layer 3 (“L3”) network) that canforward traffic between the different networks.

To connect such networks, some embodiments use a single managedinterconnection switching element or cluster thereof at the edge of eachof the segmented networks, then manage these interconnection switchingelements with a network controller that defines a logical datapath setbetween the sites. The network controller generates flow tables thatimplement the logical datapath set and passes these flow tables to theset of physical interconnection switching elements, the forwardingtables of which implement the various levels of logical flow. Ratherthan corresponding to individual end machines of the network segments,the ports of the logical switching element instead correspond to thenetwork segments themselves (e.g., a particular VLAN corresponding to aparticular port).

The managed interconnection switching elements are programmed to be ableto remove and add the local context tags (e.g., VLAN tags) of theirlocal site network. Therefore, these switching elements have the abilityto receive a local packet, strip the packet of its local context, anduse the logical switching element implementation to add a context forthe interconnecting network that identifies the interconnectionswitching element local to the destination site network for the packet.

The preceding Summary is intended to serve as a brief introduction tosome embodiments of the invention. It is not meant to be an introductionor overview of all inventive subject matter disclosed in this document.The Detailed Description that follows and the Drawings that are referredto in the Detailed Description will further describe the embodimentsdescribed in the Summary as well as other embodiments. Accordingly, tounderstand all the embodiments described by this document, a full reviewof the Summary, Detailed Description and the Drawings is needed.Moreover, the claimed subject matters are not to be limited by theillustrative details in the Summary, Detailed Description and theDrawing, but rather are to be defined by the appended claims, becausethe claimed subject matters can be embodied in other specific formswithout departing from the spirit of the subject matters.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth in the appendedclaims. However, for purpose of explanation, several embodiments of theinvention are set forth in the following figures.

FIG. 1 illustrates an example of a hierarchically-controlled network.

FIG. 2 conceptually illustrates the logical switches generated by thethree network controllers in FIG. 1.

FIG. 3 illustrates an example of such an interconnected network.

FIG. 4 conceptually illustrates the logical switching element generatedby the network controller of FIG. 3.

FIG. 5 conceptually illustrates a network architecture of someembodiments.

FIG. 6 illustrates an example of a set of managed switching elementsmanaged by a network controller.

FIG. 7 conceptually illustrates a network control system of someembodiments for managing software switching elements.

FIG. 8 illustrates an example of a network control system for managingboth physical switching elements and software switching elements.

FIG. 9 illustrates a network control system that includes networkcontrollers that control non-edge switching elements.

FIG. 10 illustrates a network that spans two data centers.

FIG. 11 illustrates two data centers with separate controller clustersthat each manage the managed switches within their respective datacenters.

FIG. 12 illustrates a managed network with a multi-level (or“federated”) implementation.

FIG. 13 illustrates an alternate implementation of a managed networkwith multiple levels of controller clusters.

FIG. 14 illustrates a data center that includes a first domain managedby a first level controller cluster and a second domain managed by adifferent first level controller cluster.

FIG. 15 conceptually illustrates a control data pipeline for ahierarchically-arranged set of network controllers at two levels thatmanage a federated network.

FIG. 16 conceptually illustrates a process performed by a second levelnetwork controller of some embodiments to generate flow entries for anevent detected at a logical control plane.

FIG. 17 conceptually illustrates a process of some embodiments thatgenerates physical control plane data from the 1L forwarding plane datareceived at the 1L controller from the 2L controller.

FIG. 18 conceptually illustrates these input and output tables throughthe various flow generation operations of some embodiments.

FIG. 19 illustrates a set of logical datapath sets for an examplefederated network of some embodiments.

FIG. 20 conceptually illustrates the path of a packet through fourmanaged switches between its source machine in a first domain and itsdestination machine in a second domain.

FIG. 21 conceptually illustrates in greater detail a process of someembodiments for processing packets by a first hop managed switchingelement in a federated network.

FIG. 22 conceptually illustrates in greater detail a process of someembodiments for processing packets by an interconnection managedswitching element for a packet exiting the domain of the interconnectionmanaged switching element in a federated network.

FIG. 23 conceptually illustrates in greater detail a process of someembodiments for processing packets by an interconnecting managedswitching element for a packet entering the domain of theinterconnecting managed switching element in a federated network.

FIG. 24 conceptually illustrates a different view of the processingperformed by a source managed switching element.

FIG. 25 illustrates a processing pipeline for a specific type of networkperforming OSI layer 2 forwarding (e.g., forwarding based on MACaddress).

FIG. 26 illustrates a network with three separate data centers.

FIG. 27 illustrates three 1L logical datapath sets connected by a 2Llogical datapath set, along with some of the port mappings performedduring packet processing by the logical switching elements asimplemented in the managed switching elements of the network shown inFIG. 26.

FIG. 28 conceptually illustrates a process of some embodiments forsetting a flag upon receiving a packet.

FIG. 29 conceptually illustrates a process of some embodiments fordetermining whether to broadcast a packet to all ports of a first levellogical datapath set.

FIG. 30 conceptually illustrates a network in which various 1L domainsare not fully connected.

FIG. 31 illustrates a network with four data centers with three levelsof network controller clusters.

FIG. 32 conceptually illustrates three levels of logical switchesimplemented for the network of FIG. 31, as well as some of the mappingsbetween the ports of these logical switches.

FIG. 33 conceptually illustrates network that allows the first leveldatapath to be sliced into multiple second level datapaths.

FIG. 34 conceptually illustrates example logical switches for thenetwork in FIG. 33.

FIG. 35 conceptually illustrates three separate segmented networks.

FIG. 36 conceptually illustrates a solution for the networks in FIG. 35that locates an interconnection switching element at the edge of each ofthe segmented networks, then manages these interconnection switchingelements with a network controller cluster.

FIG. 37 conceptually illustrates a logical switching element defined bythe network controller cluster in FIG. 36 and implemented by the threeinterconnection managed switching elements in FIG. 36.

FIG. 38 conceptually illustrates information stored in a managedswitching element for interconnecting segmented networks.

FIG. 39 illustrates a scenario for the networks in FIG. 35 in which twodifferent network controllers generate flow entries for two differentlogical networks, and push the flows to the same switching elements.

FIG. 40 conceptually illustrates logical switching elements defined bythe network controller clusters, respectively, and implemented by thethree interconnecting managed switching elements.

FIG. 41 conceptually illustrates information stored in a managedswitching element for interconnecting the segmented networks on the twological switching elements.

FIG. 42 conceptually illustrates four segmented networks connected usingmultiple logical layers.

FIG. 43 conceptually illustrates the three logical datapath sets definedby the three network controller clusters of FIG. 42.

FIG. 44 conceptually illustrates a process of some embodiments performedby the network controller for an interconnecting network in order togenerate new flow entries for an event detected at the logical controlplane.

FIG. 45 conceptually illustrates input and output tables through thevarious flow generation operations of some embodiments.

FIG. 46 conceptually illustrates optimization processing in ahigher-level network controller of some embodiments.

FIG. 47 conceptually illustrates the path of a packet through twomanaged switching elements between its source in a first network segmentand its destination in a second network segment.

FIG. 48 conceptually illustrates a process of some embodiments forprocessing packets by a source network's interconnection switchingelement.

FIG. 49 conceptually illustrates an example of some of the forwardingtable operations performed by a source interconnection switchingelement.

FIG. 50 conceptually illustrates in greater detail a process of someembodiments for processing packets by a destination network'sinterconnection switching element.

FIG. 51 conceptually illustrates an example of some of the forwardingtable operations performed by a destination interconnection switch.

FIG. 52 conceptually illustrates a more complex network, with fourseparate 1L domains of three interconnected networks each.

FIG. 53 illustrates an example of a packet traveling through a networkfrom a first VM to a second VM.

FIG. 54 conceptually illustrates a computer system with which someembodiments of the invention are implemented.

DETAILED DESCRIPTION

In the following detailed description of the invention, numerousdetails, examples, and embodiments of the invention are set forth anddescribed. However, it will be clear and apparent to one skilled in theart that the invention is not limited to the embodiments set forth andthat the invention may be practiced without some of the specific detailsand examples discussed.

Some embodiments of the invention provide a network control system thatallows several different logical datapath sets to be specified forseveral different users through one or more shared forwarding elementswithout allowing the different users to control or even view eachother's forwarding logic. The shared forwarding elements in someembodiments can include virtual or physical network switches, softwareswitches (e.g., Open vSwitch), routers, and/or other switching devices,as well as any other network elements (such as load balancers, etc.)that establish connections between these switches, routers, and/or otherswitching devices. Such forwarding elements (e.g., physical switches orrouters) are also referred to below as switching elements. In contrastto an off the shelf switch, a software forwarding element is a switchingelement that in some embodiments is formed by storing its switchingtable(s) and logic in the memory of a standalone device (e.g., astandalone computer or a device (e.g., a computer) that also executes ahypervisor and one or more virtual machines on top of that hypervisor.

More specifically, the network control system of some embodimentsmanages networks over which machines (e.g. virtual machines) belongingto several different users (i.e., several different users in a privateor public hosted environment with multiple hosted computers and managedforwarding elements that are shared by multiple different related orunrelated tenants) may exchange data packets for separate logicaldatapath sets. That is, machines belonging to a particular user mayexchange data with other machines belonging to the same user over alogical datapath set for that user, while machines belonging to adifferent user exchange data with each other over a different logicaldatapath set implemented on the same physical managed network. In someembodiments, a logical datapath set (also referred to as a logicalforwarding element, logical switching element (e.g., logical switch,logical router), or logical network in some cases) is a logicalconstruct that provides switching fabric to interconnect several logicalports, to which a particular user's machines (physical or virtual) mayattach.

These managed, shared switching elements are referred to below asmanaged switching elements or managed forwarding elements as they aremanaged by the network control system in order to implement the logicaldatapath sets. In some embodiments described below, the control systemmanages these switching elements by pushing physical control plane datato them, as further described below. Switching elements generallyreceive data (e.g., a data packet) and perform one or more processingoperations on the data, such as dropping a received data packet, passinga packet that is received from one source device to another destinationdevice, processing the packet and then passing it to a destinationdevice, etc. In some embodiments, the physical control plane data thatis pushed to a switching element is converted by the switching element(e.g., by a general purpose processor of the switching element) tophysical forwarding plane data that specify how the switching element(e.g., how a specialized switching circuit of the switching element)processes data packets that it receives.

In some embodiments, the network control system includes one or morenetwork controllers (also called controller instances below) that allowthe system to accept logical datapath sets from users and to configurethe switching elements to implement these logical datapath sets. Thesecontrollers allow the system to virtualize control of the sharedswitching elements and the logical networks that are defined by theconnections between these shared switching elements, in a manner thatprevents the different users from viewing or controlling each other'slogical datapath sets and logical networks while sharing the sameswitching elements.

In some embodiments, each controller instance is a device (e.g., ageneral-purpose computer) that executes one or more modules thattransform the user input from a logical control plane to a logicalforwarding plane, and then transform the logical forwarding plane datato physical control plane data. These modules in some embodimentsinclude a control module and a virtualization module. A control moduleallows a user to specify and populate logical datapath sets, while avirtualization module implements the specified logical datapath sets bymapping the logical datapath sets onto the physical switchinginfrastructure. In some embodiments, the control and virtualizationmodules are two separate applications, while in other embodiments theyare part of the same application.

In some embodiments described below, the virtualization module convertslogical forwarding plane data directly to physical control plane datathat is pushed to the managed switches. While some embodiments performthis conversion directly from the logical forwarding plane to thephysical control plane (providing data that is customized specificallyfor each managed switching element), other embodiments improve thescalability of the system by introducing an intermediate universalforwarding state that provides data at the physical control plane levelthat is not customized to any particular managed switching element.

In such embodiments, from the logical forwarding plane data for aparticular logical datapath set, the virtualization module of someembodiments generates universal physical control plane (UPCP) data thatis generic for any managed switching element that implements the logicaldatapath set. In some embodiments, this virtualization module is part ofa controller instance that is a master controller for the particularlogical datapath set. This controller is referred to as the logicalcontroller.

In some embodiments, the UPCP data is then converted to customizedphysical control plane (CPCP) data for each particular managed switchingelement by a controller instance that is a master physical controllerinstance for the particular managed switching element, or by a chassiscontroller for the particular managed switching element, as furtherdescribed in U.S. patent application Ser. No. 13/589,077, now issued asU.S. Pat. No. 9,178,833, entitled “Chassis Controller,” and filed onAug. 17, 2012. When the chassis controller generates the CPCP data, thechassis controller obtains the UPCP data from the virtualization moduleof the logical controller through the physical controller.

Irrespective of whether the physical controller or chassis controllergenerates the CPCP data, the CPCP data for a particular managedswitching element needs to be propagated to the managed switchingelement. In some embodiments, the CPCP data is propagated through anetwork information base (NIB) data structure, which in some embodimentsis an object-oriented data structure. Several examples of using the NIBdata structure are described in U.S. patent application Ser. No.13/177,529, now issued as U.S. Pat. No. 8,743,889, and Ser. No.13/177,533, now issued as U.S. Pat. No. 8,817,620, which areincorporated herein by reference. As described in these applications,the NIB data structure is also used in some embodiments to may serve asa communication medium between different controller instances, and tostore data regarding the logical datapath sets (e.g., logical switchingelements) and/or the managed switching elements that implement theselogical datapath sets.

However, other embodiments do not use the NIB data structure topropagate CPCP data from the physical controllers or chassis controllersto the managed switching elements, to communicate between controllerinstances, and to store data regarding the logical datapath sets and/ormanaged switching elements. For instance, in some embodiments, thephysical controllers and/or chassis controllers communicate with themanaged switching elements through OpenFlow entries and updates over theconfiguration protocol. Also, in some embodiments, the controllerinstances use one or more direct communication channels (e.g., RPCcalls) to exchange data. In addition, in some embodiments, thecontroller instances (e.g., the control and virtualization modules ofthese instance) express the logical and/or physical data in terms ofrecords that are written into the relational database data structure. Insome embodiments, this relational database data structure is part of theinput and output tables of a table mapping engine (called nLog) that isused to implement one or more modules of the controller instances.

Several embodiments described below provide network control systems thatcompletely separate the logical forwarding space (i.e., the logicalcontrol and forwarding planes) from the physical forwarding space (i.e.,the physical control and forwarding planes). These control systemsachieve such separation by using a mapping engine to map the logicalforwarding space data to the physical forwarding space data. Bycompletely decoupling the logical space from the physical space, thecontrol systems of such embodiments allow the logical view of thelogical forwarding elements to remain unchanged while changes are madeto the physical forwarding space (e.g., virtual machines are migrated,physical switches or routers are added, etc.).

In some embodiments, the network control system manages more than asingle network. The network control system of some embodiments providesdifferent mechanisms for connecting two or more networks together inorder to achieve communication between machines (e.g., virtual machinesor physical machines) located within the different networks. In someembodiments, the networks connected together are located at differentsites (e.g., different data centers), each of which consists of severalend machines. Some embodiments utilize interconnection switchingelements located at the edges of the different networks that arespecifically programmed to enable packets from a source network to beread by a destination network. In some embodiments, the connection ofthe networks forms a larger logical network that includes the individualsite networks.

In some embodiments, the individual networks logically connectedtogether are themselves logical networks. Each logical network ismanaged by a network control system that enables the specification of alogical datapath set for the logical network. The network control systemconfigures a set of shared switching elements to implement the specifiedlogical datapath set, thereby virtualizing the switching elements. Inorder to interconnect multiple logical networks, some embodimentsprovide a hierarchical control system that includes a hierarchicalarrangement of network controllers. A set of higher-level networkcontrollers receives a specification of a higher-level logical datapathset that includes the machines of all of the lower-level logicalnetworks. The higher-level network controllers generate flow entriesthat are passed down to the lower-level network controllers. Theselower-level network controllers configure the shared switching elementswithin their respective networks in order to implement the higher-levellogical datapath set on top of their own respective lower-level logicaldatapath sets. In this case, the higher-level logical datapath setserves to interconnect the lower-level logical datapath sets, therebyinterconnecting the networks.

The hierarchical controller arrangement in turn results in hierarchicalprocessing of packets by the switching elements within the network. Asstated, the higher-level network controllers generate flow tables thatimplement the highest-level logical datapath set that governs packetforwarding between end machines of the network, then pass these tablesdown to the lower-level network controllers. The lower-level networkcontrollers then incorporate the received higher-level flow entries intotheir own flow tables that implement the lower-level logical datapathsets. Each set of lowest-level network controllers generates flow tablesfor the physical managed switching elements within the network of thecontroller set, and passes these generated flow tables to the physicalswitching elements. These flow tables specify lookup entries that themanaged switching elements use to process packets within the logicalnetwork. When a managed switching element receives a packet on aphysical port, the managed switching element maps the packet to logicalports of the hierarchical data paths, makes a forwarding decision to anegress port at the highest level, and maps the identified egress portback down through the hierarchy of data paths to a physical egress port.

FIG. 1 illustrates an example of such a hierarchically-controllednetwork 100. This figure illustrates a first data center 105 and asecond data center 110. Located at each of the data centers are four endmachines, which could be virtual machines or physical machines,depending on the setups of the particular data centers. These endmachines are connected to managed switching elements, which in someembodiments are physical switching elements that implement logicaldatapath sets (also referred to as logical switching elements) generatedby the network controllers. The managed switching elements includesoftware switching elements, dedicated hardware switching elements, or acombination thereof in different embodiments. As shown, a managedswitching element 135 at the first data center 105 and a managedswitching element 140 at the second data center 110 connect through anexternal network 130 (e.g., via a tunnel). These two managed switchingelements 135 and 140 function as interconnecting switching elements forthe networks.

Additionally, the first data center 105 includes a first level networkcontroller 115 while the second data center 110 includes a differentfirst level network controller 120. The first level network controller115 generates a first level logical switching element that connects thefour machines located within data center 105, while the first levelnetwork controller 120 generates a first level logical switching elementthat connects the four machines located within the data center 110.These logical switching elements are conceptually illustrated in FIG. 2,described below.

In addition to the network controllers shown within the data centers 105and 110, the network 100 includes a second level network controller 125.The network controller 125 generates a second level logical switchingelement that connects together the logical switching elements generatedby the first level network controllers 115 and 120. In this way, asingle point of control can be used to managed the entire network 100,while the networks in each of the separate data centers 105 and 110 canoperate on their own, should their connection to each other or thesecond level network controller 125 be disrupted.

As mentioned, FIG. 2 conceptually illustrates the logical switchingelements generated by the three network controllers in FIG. 1, andimplemented by the managed switching elements. A first logical switchingelement 205 for the first data center 105 has five logical ports, fourof which connect to the machines in the data center. The fifth logicalport connects to the remote machines at the second data center 110(logically connecting to a port on the logical switching element 210).The second logical switching element 210 for the second data center 110is similarly arranged, with four ports connecting to machines at thelocal data center and a fifth port connecting to the remote machines atthe first data center 105 (logically connecting to a port on the logicalswitching element 205). The second level logical switching element 225includes eight ports for logically connecting to the eight machines(through the logical ports at the first level logical switchingelements).

When a machine in the first data center 105 sends a packet to a machinein the second data center 110, these logical switching elements areimplemented by the physical switching elements of the network in orderto transport the packet. The first managed switching element thatreceives the packet initially performs ingress context mappingoperations to identify the ingress port on the logical switching element205 that corresponds to the physical ingress port from which the packetis received. The switching element then maps this identified first levellogical ingress port to a second level logical ingress port of thesecond level logical switching element 225. The flow tables in themanaged switching element implementing the second level logicalswitching element use the packet destination information to make aforwarding decision to a second level egress port, which is then mappedto a logical egress port in the logical switching element 210 thatcorresponds to the destination machine. The generation and operation ofthese logical switching elements will be described in detail in thesections below.

The above description relates to the situation in which the networkslogically connected together are all managed networks governed bylogical controllers that implement logical switching elements within themanaged switching elements of the networks, and use those same managedswitching elements to implement the interconnection between thenetworks. In some embodiments, the networks logically connected togethermay include unmanaged segmented networks. In some cases, a networkmanager will want to provide connections between a first networksegmented using a first tagging or tunneling technique (e.g., VLAN,Mac-in-Mac, L2 over L3, MPLS, etc.) and a second network segmented usinga second tagging or tunneling technique. Even when the two networks usethe same technique (e.g., both networks using VLANs), the implementationof that technique (e.g., the structure of the tags used in packetheaders) may be different between the two networks such that they areeffectively using two different techniques. Some embodiments provide amechanism for connecting such differently-segmented networks across acommon interconnecting network (e.g., an L3 network) that can forwardtraffic between the different networks.

To connect such networks, some embodiments use a single managedinterconnection switching element (e.g., an extender) or cluster thereofat the edge of each of the segmented networks, then manage theseinterconnection switching elements with a network controller thatdefines a logical datapath set between the sites. The network controllergenerates flow tables that implement the logical datapath set and passesthese flow tables to the set of physical interconnection switchingelements, the forwarding tables of which implement the various levels oflogical flow. Rather than corresponding to individual end machines ofthe network segments, the ports of the logical switching element insteadcorrespond to the network segments themselves (e.g., a particular VLANcorresponding to a particular port).

The managed interconnection switching elements are programmed to be ableto remove and add the local context tags (e.g., VLAN tags) of theirlocal site network. Therefore, these switching elements have the abilityto receive a local packet, strip the packet of its local context, anduse the logical switching element implementation to add a context forthe interconnecting network that identifies the interconnectionswitching element local to the destination site network for the packet.

FIG. 3 illustrates an example of such an interconnected network 300.This figure illustrates four segmented site networks 305-320. Located ateach of the site networks are several machines belonging to a networksegment (e.g., VLAN, MPLS label, etc.) at that site network. In each ofthese networks, machines on one network segment can communicate withother machines on the same network segment (e.g., machine A and machineB). However, without the provision of interconnecting services, machineson a segment of site network 305 cannot communicate with machines on asegment of one of the other site networks (e.g., machine A and machine Gcannot communicate).

In this situation, however, interconnection switching elements 325-340are located at the edge of each site network 305-320, connecting thesite networks to an interconnecting physical network 345. Thisinterconnecting physical network acts as a common substrate for the foursite networks (i.e., all four site networks attach to the network 345).As shown, a network controller 350 connects to each of the fourinterconnection switching elements 325-340. The network controller 350(which may be a controller cluster or a single controller instance)generates flow tables for implementing a logical switching element thatconnects the four site networks, then passes the flow tables to the fourinterconnection switching elements 225-340.

FIG. 4 conceptually illustrates this logical switching element 400generated by the network controller 350. The logical switching element400 includes four ports, one for each network segment connected by theswitching element. The segments include a first segment 405 located atthe first segmented network 305, a second segment 410 located at thesecond segmented network 310, a third segment 415 located at the thirdsegmented network 315, and a fourth segment 420 located at the fourthsegmented network 320. Whereas the ports in the logical switchingelement of FIG. 2 above correspond to specific machines, here the portscorrespond to network segments (e.g., VLANs).

The above FIG. 3 illustrates a managed network between the sites thathas a single network controller at a single level. In addition, someembodiments enable the site networks to be grouped via a hierarchicalstructure such as that described above for FIG. 1. That is, several sitenetworks are interconnected with a first lower-level network controller,while several additional site networks are interconnected with a secondlower-level network controller. These two groups of site networks canthen be themselves interconnected with a higher-level network controllerthat communicates with the lower-level network controller.

FIGS. 2-4 illustrate examples of the interconnection of site networks.Several more detailed embodiments are described below. First, Section Idescribes the environment of a managed network of end machines of someembodiments. Section II then describes the hierarchical use of networkcontrollers to create a hierarchical logical network. Next, Section IIIdescribes the interconnection of unmanaged network segments through theuse of both single-level and hierarchical logical interconnectionnetworks. Finally, Section IV describes an electronic system with whichsome embodiments of the invention are implemented.

I. Environment

The following section will describe the environment in which someembodiments of the inventions are implemented. In the presentapplication, switching elements and machines may be referred to asnetwork elements, switches, or other terms. In addition, a networkmanaged by one or more network controllers may be referred to as amanaged network in the present application. In some embodiments, themanaged network includes only managed switching elements (e.g.,switching elements that are controlled by one or more networkcontrollers) while, in other embodiments, the managed network includesmanaged switching elements as well as unmanaged switching elements(e.g., switching elements that are not controlled by a networkcontroller).

FIG. 5 conceptually illustrates a network architecture 500 of someembodiments. As shown, the network architecture 500 includes networkcontrollers 510 and 520, managed switching elements 530-550, andmachines 555-585.

In some embodiments, the managed switching elements 530-550 route (i.e.,process and/or forward) network data (e.g., packets) between networkelements in the network that are coupled to the managed switchingelements 530-550. For instance, the managed switching element 530 routesnetwork data between the machines 555-565 and the managed switchingelement 540. Similarly, the managed switching element 540 routes networkdata between the machine 570 and the managed switching elements 530 and550, and the managed switching element 550 routes network data betweenthe machines 575-585 and the managed switching element 540.

The managed switching elements 530-550 of some embodiments can beconfigured to route network data according to defined rules. In someembodiments, the managed switching elements 530-550 route network databased on routing criteria defined in the rules. Examples of routingcriteria include source media access control (MAC) address, destinationMAC address, packet type, source Internet Protocol (IP) address,destination IP address, source port, destination port, and/or networksegment identifier (e.g., virtual local area network (VLAN) identifier,multi-protocol label switching (MPLS) label, etc.), among other routingcriteria.

In some embodiments, the managed switching elements 530-550 can includestandalone physical switching elements, software switching elements thatoperate within a computer, or any other type of switching element. Forexample, each of the managed switching elements 530-550 may beimplemented as a hardware switching element, a software switchingelement, a virtual switching element, a network interface controller(NIC), or any other type of network element that can route network data.Moreover, the software or virtual switching elements may operate on adedicated computer, or on a computer that also performs non-switchingoperations.

The machines 555-585 send and receive network data between each otherover the network (and, in some cases, with other machines outside thenetwork). In some embodiments, the machines 555-585 are referred to asnetwork hosts that are each assigned a network layer host address (e.g.,IP address). In some cases, the machines 555-585 are referred to as endsystems because the machines 555-585 are located at the edge of thenetwork. In some embodiments, each of the machines 555-585 can be adesktop computer, a laptop computer, a smartphone, a virtual machine(VM) running on a computing device, a terminal, or any other type ofnetwork host.

In some embodiments, each of the network controllers 510 and 520controls one or more managed switching elements 530-550 that are locatedat the edge of a network (e.g., edge switching elements or edgedevices). In this example, the managed switching elements 530-550 areedge switching elements. That is, the managed switching elements 530-550are switching elements that are located at or near the edge of thenetwork. In some embodiments, an edge switching element is the lastswitching element before one or more end machines (the machines 555-585in this example) in a network. That is, an edge switching element is thefirst switching element that receives network data sent from one or moreend machines and is the last switching element that receives networkdata sent to the one or more end machines. As indicated by dashed arrowsin FIG. 5, the network controller 510 controls (i.e., manages) switchingelements 530 and 540 and the network controller 520 controls switchingelement 550. In this application, a switching element that is controlledby a network controller of some embodiments may be referred to as amanaged switching element.

In addition to controlling edge switching elements, the networkcontrollers 510 and 520 of some embodiments also utilize and controlnon-edge switching elements (e.g., pool nodes and extenders, which aredescribed in further detail below) that are inserted in the network tosimplify and/or facilitate the operation of the managed edge switchingelements. For instance, in some embodiments, the network controllers 510and 520 require that the managed switching elements be interconnected ina hierarchical switching architecture that has several edge switchingelements as the leaf nodes in the hierarchical switching architectureand one or more non-edge switching elements as the non-leaf nodes inthis architecture. In some such embodiments, each edge switching elementconnects to one or more of the non-leaf switching elements, and usessuch non-leaf switching elements to facilitate the communication of theedge switching element with other edge switching elements. Examples ofsuch communications with an edge switching elements in some embodimentsinclude (1) routing of a packet with an unknown destination address(e.g., unknown MAC address) to the non-leaf switching element so thatthe non-leaf switching element can route the packet to the appropriateedge switching element, (2) routing a multicast or broadcast packet tothe non-leaf switching element so that the non-leaf switching elementcan distribute the multicast or broadcast packet to the desireddestinations, and (3) routing packets to destination machines externalto the network 500.

Some embodiments employ one level of non-leaf (non-edge) switchingelements that connect to edge switching elements and in some cases toother non-leaf switching elements. Other embodiments, on the other hand,employ multiple levels of non-leaf switching elements, with each levelof non-leaf switching elements after the first level serving as amechanism to facilitate communication between lower level non-leafswitching elements and leaf switching elements. In some embodiments, thenon-leaf switching elements are software switching elements that areimplemented by storing forwarding tables in the memory of a standalonecomputer instead of an off the shelf switch. In some embodiments, thestandalone computer may also be executing a hypervisor and one or morevirtual machines on top of that hypervisor. Irrespective of the mannerby which the leaf and non-leaf switching elements are implemented, thenetwork controllers 510 and 520 of some embodiments store switchingstate information regarding the leaf and non-leaf switching elements.

As mentioned above, the switching elements 530-550 of some embodimentsroute network data between network elements in the network. In someembodiments, the network controllers 510 and 520 configure the routingof network data between the network elements in the network by themanaged switching elements 530-550. In this manner, the networkcontrollers 510 and 520 can control the flow (i.e., specify the datapath) of network data between network elements.

For example, the network controller 510 might instruct the managedswitching elements 530 and 540 to route network data from the machine555 to the machine 570 (and vice versa) and to not route (e.g., drop)network data from other machines to the machines 555 and 570. In suchcase, the network controller 510 controls the flow of network datathrough the managed switching elements 530 and 540 such that networkdata transmitted to and from the machine 555 is only routed to themachine 570. Thus, the machines 555 and 570 cannot send and receivenetwork data to and from the machines 560, 565, and 575-585.

In some embodiments, the network controllers 510 and 520 store physicalnetwork information and logical network information. The physicalnetwork information specifies the physical components in the managednetwork and how the physical components are physically connected oneanother in the managed network. For example, the physical networkinformation may include the number of machines, managed switchingelements, pool nodes, and extenders (the latter two are described infurther detail in the following sections), and how the components arephysically connected to one another in the managed network. The logicalnetwork information may specify the logical connections between a set ofphysical components in the managed network (e.g., machines) and amapping of the logical connections across the physical components of themanaged network.

Some embodiments of the network controllers 510 and 520 implement alogical switching element across the managed switching elements 530-550based on the physical network information and the logical switchingelement information described above. A logical switching element can bedefined to function any number of different ways that a switchingelement might function. The network controllers 510 and 520 implementthe defined logical switching element through control of the managedswitching elements 530-550. In some embodiments, the network controllers510 and 520 implement multiple logical switching elements across themanaged switching elements 530-550. This allows multiple differentlogical switching elements to be implemented across the managedswitching elements 530-550 without regard to the network topology of thenetwork.

In some embodiments, a logical datapath set (LDPS) defines a logicalswitching element. A logical datapath set, in some embodiments, is a setof network data paths through the managed switching elements 530-550that implement the logical switching element and the logical switch'sdefined functionalities. In these embodiments, the network controllers510 and 520 translate (e.g., map) the defined logical datapath set intonetwork configuration information for implementing the logical switchingelement. The network controllers 510 and 520 translate the definedlogical datapath set into a corresponding set of data flows (i.e., datapaths) between network elements in the network, in some embodiments. Inthese instances, the network controllers 510 and 520 instruct themanaged switching elements 530-550 to route network data according tothe data flows and, thus, implement the functionalities of the definedlogical switching element. Within this application, logical datapathset, logical switch, logical switching element, and logical datapath maybe used interchangeably.

Different embodiments of the network controllers 510 and 520 areimplemented differently. For example, some embodiments implement thenetwork controllers 510 and 520 in software as instances of a softwareapplication. In these cases, the network controllers 510 and 520 may beexecuted on different types of computing devices, such as a desktopcomputer, a laptop computer, a smartphone, etc. In addition, thesoftware application may be executed on a virtual machine that runs on acomputing device in some embodiments. In some embodiments, the networkcontrollers 510 and 520 are implemented in hardware (e.g., circuits). Insome embodiments, the network controllers 510 and 520 communicate witheach other, in order to distribute information. In fact, in someembodiments, the network controllers that govern a logical network actas a controller cluster that behaves as a single network controllerdistributed across multiple machines.

As mentioned above by reference to FIG. 5, the managed switchingelements controlled by network controllers of some embodiments may bephysical switching elements. FIG. 6 illustrates an example of a networkcontrol system that includes physical switching elements. This figureconceptually illustrates a network control system 600 of someembodiments for managing physical switching elements. Specifically, thenetwork control system 600 manages network data in a data center thatincludes top of the rack (TOR) switching elements 630-650 and racks ofhosts 660-680. Network controllers 610 and 620 manage the network bycontrolling the TOR switching elements 630-650.

A TOR switching element, in some embodiments, routes network databetween hosts in the TOR switch's rack and network elements coupled tothe TOR switching element. In the example illustrated in FIG. 6, the TORswitching element 630 routes network data between the rack of hosts 660and TOR switching elements 640 and 650, the TOR switching element 640routes network data between the rack of hosts 670 and TOR switchingelements 630 and 650, and the TOR switching element 650 routes networkdata between the rack of hosts 680 and TOR switching elements 630 and640.

As shown, each rack of hosts 660-680 includes multiple hosts. The hostsof some embodiments in the racks of hosts 660-680 are physical computingdevices. In some embodiments, each host is a computing device that isassigned a network layer host address (e.g., IP address). The hosts ofsome embodiments send and receive network data to and from each otherover the network.

As mentioned above, the network controller of some embodiments can beimplemented in software as an instance of an application. As illustratedin FIG. 6, the network controllers 610 and 620 are instances of asoftware application. As shown, each of the network controllers 610 and620 includes several software layers: a control application layer, avirtualization application layer, and a networking operating systemlayer.

In some embodiments, the control application layer receives input (e.g.,from a user) that specifies a network switching element. The controlapplication layer may receive the input in any number of differentinterfaces, such as a graphical user interface (GUI), a command lineinterfaces, a web-based interface, a touchscreen interface, through anapplication programming interface (API) exposed to other networkcontroller instances, etc. In some embodiments, the input specifiescharacteristics and behaviors of the network switching element, such asthe number of switching element ports, access control lists (ACLs),network data forwarding, port security, or any other network switchingelement configuration options.

The control application layer of some embodiments defines a logicaldatapath set based on user input that specifies a network switchingelement. As noted above, a logical datapath set is a set of network datapaths through managed switching elements that are used to implement theuser-specified network switching element. In other words, the logicaldatapath set is a logical representation of the network switchingelement and the network switch's specified characteristics andbehaviors.

Some embodiments of the virtualization application layer translate thedefined logical datapath set into network configuration information forimplementing the logical network switching element across the managedswitching elements in the network. For example, the virtualizationapplication layer of some embodiments translates the defined logicaldatapath set into a corresponding set of data flows. In some of thesecases, the virtualization application layer may take into accountvarious factors (e.g., logical switching elements that are currentlyimplemented across the managed switching elements, the current networktopology of the network, etc.), in determining the corresponding set ofdata flows.

The network operating system layer of some embodiments configures themanaged switching elements' routing of network data. In someembodiments, the network operating system instructs the managedswitching elements to route network data according to the set of dataflows determined by the virtualization application layer.

In some embodiments, the network operating system layer maintainsseveral views of the network based on the current network topology. Oneview that the network operating system layer of some embodimentsmaintains is a logical view. The logical view of the network includesthe different logical switching elements that are implemented across themanaged switching elements, in some embodiments. Some embodiments of thenetwork operating system layer maintain a managed view of the network.Such managed views include the different managed switching elements inthe network (i.e., the switching elements in the network that thenetwork controllers control). In some embodiments, the network operatingsystem layer also maintains relationship data that relate the logicalswitching elements implemented across the managed switching elements tothe managed switching elements.

The network controller of some embodiments is described in greaterdetail in U.S. application Ser. No. 13/177,533, filed on Jul. 6, 2011and entitled “Network Virtualization Apparatus and Method”, which isincorporated herein by reference. As explained in further detail in thisapplication, the control application of some embodiments performs atable-mapping operation (e.g., using an nLog table mapping engine) totransform tables in a logical control plane to tables in a logicalforwarding plane. The logical control plane, in some embodiments,includes a collection of constructs that allow the control applicationand its users to specify one or more logical datapath sets within thelogical control plane. The logical forwarding plane, in someembodiments, includes one or more data path sets of one or more users.Thus, the control application transforms the collection of constructs(e.g., as specified by a user) into the logical datapath sets. Thevirtualization application transforms logical forwarding plane data intophysical control plane data (e.g., also using an nLog table mappingengine) that can be pushed down to the physical managed switchingelements.

While FIG. 6 (and other figures in this application) may show a set ofmanaged switching elements managed by a network controller, someembodiments provide several network controllers (also referred to as acluster of network controllers or a control cluster) for managing theset of managed switching elements. In other embodiments, differentcontrol clusters may manage different sets of managed switchingelements. Employing a cluster of network controllers in such embodimentsto manage a set of managed switching elements increases the scalabilityof the managed network and increases the redundancy and reliability ofthe managed network. In some embodiments, the network controllers in acontrol cluster share (e.g., through the network operating system layerof the network controllers) data related to the state of the managednetwork in order to synchronize the network controllers.

FIG. 7 conceptually illustrates a network control system 700 of someembodiments for managing software switching elements. As shown, thenetwork control system 700 includes network controllers 710 and 720, TORswitching elements 730-750, and racks of hosts 760-780.

The TOR switching elements 730-750 are similar to the TOR switchingelements 630-650. The TOR switching elements 730-750 route network databetween network elements in the network that are coupled to the TORswitching elements 730-750. In this example, the TOR switching element730 routes network data between the rack of hosts 760 and TOR switchingelements 740 and 750, the TOR switching element 740 routes network databetween the rack of hosts 770 and TOR switching elements 730 and 750,and the TOR switching element 750 routes network data between the rackof hosts 780 and TOR switching elements 730 and 740. Since the TORswitching elements 730-750 are not managed switching elements, thenetwork controllers 710 and 720 do not control these switching elements.Thus, the TOR switching elements 730-750 rely on the switching elements'preconfigured functionalities to route network data.

As illustrated in FIG. 7, each host in the racks of hosts 760-780includes a software switching element (an open virtual switch (OVS) inthis example) and several VMs. The VMs are virtual machines that areeach assigned a set of network layer host addresses (e.g., a MAC addressfor network layer 2, an IP address for network layer 3, etc.) and cansend and receive network data to and from other network elements overthe network.

The OVSs of some embodiments route network traffic between networkelements coupled to the OVSs. For example, in this example, each OVSroutes network data between VMs that are running on the host on whichthe OVS is running, OVSs running on other hosts in the rack of hosts,and the TOR switching element of the rack.

By running a software switching element and several VMs on a host, thenumber of end machines or network hosts in the network may increase.Moreover, when a software switching element and several VMs are run onhosts in the racks of hosts 760-780, the network topology of the networkis changed. In particular, the TOR switching elements 730-750 are nolonger edge switching elements. Instead, the edge switching elements inthis example are the software switching elements running on the hostssince these software switching elements are the last switching elementsbefore end machines (i.e., VMs in this example) in the network. Whilethe examples of software switching elements are open virtual switches(OVSs) in this example, in some embodiments different types of softwareswitching elements might be used in order to implement the logicalswitching elements.

The network controllers 710 and 720 perform similar functions as thenetwork controllers 610 and 620, described above by reference to FIG. 6,and also are for managing edge switching elements. As such, the networkcontrollers 710 and 720 manage the software switches that are running onthe hosts in the rack of hosts 760-780.

The above FIGS. 6 and 7 illustrate a network control system for managingphysical switching elements and a network control system for managingsoftware switching elements, respectively. However, the network controlsystem of some embodiments can manage both physical switching elementsand software switching elements. FIG. 8 illustrates an example of such anetwork control system. In particular, this figure conceptuallyillustrates a network control system 800 of some embodiments formanaging TOR switching element 830 and software switching elementsrunning on hosts in the racks of hosts 870 and 880.

The network controllers 810 and 820 perform similar functions as thenetwork controllers 610 and 620, which described above by reference toFIG. 6, and also are for managing edge switching elements. In thisexample, the managed switching element 830 and the software switchingelements running on the hosts in the racks of hosts 870 and 880 are edgeswitching elements because they are the last switching elements beforeend machines in the network. In particular, the network controller 810manages the TOR switching element 830 and the OVSs that are running onthe hosts in the rack of hosts 870, and the network controller 820manages the OVSs that are running on the hosts in the rack of hosts 880.

The above figures illustrate examples of network controllers thatcontrol edge switching elements in a network. However, in someembodiments, the network controllers can control non-edge switchingelements as well. FIG. 9 illustrates a network control system thatincludes such network controllers. In particular, FIG. 9 conceptuallyillustrates a network control system 900 of some embodiments formanaging TOR switching elements 930-950 and OVSs running on hosts in theracks of hosts 970 and 980.

As shown in FIG. 9, the network controllers 910 and 920 manage edgeswitching elements and non-edge switching elements. Specifically, thenetwork controller 910 manages the TOR switching elements 930 and 940,and the software switches running on the hosts in the rack of hosts 970.The network controller 920 manages TOR switching element 950 and thesoftware switching elements running on the hosts in the rack of hosts980. In this example, the TOR switching element 930 and the softwareswitching elements running on the hosts in the racks of hosts 970 and980 are edge switching elements, and the TOR switching elements 940 and950 are non-edge switching elements. The network controllers 910 and 920perform similar functions as the network controllers 610 and 620, whichare described above by reference to FIG. 6.

II. Hierarchical Network Controller Structure

In some embodiments, several individual logical networks (such as thoseshown above in FIGS. 5-9) can be logically connected together to form alarger multi-level logical network. In order to interconnect multiplelogical networks, some embodiments provide a hierarchical networkcontrol system that includes a hierarchical arrangement of networkcontrollers. A set of higher-level network controllers receives aspecification of a higher-level logical datapath set that includes themachines of all of the lower-level logical networks to beinterconnected. The higher-level network controllers generate flowentries that are passed down to the lower-level network controllers.These lower-level network controllers modify the flow entries toincorporate lower-level logical datapath set information, then configurethe managed switching elements within their respective networks in orderto implement the higher-level logical datapath set on top of their ownrespective lower-level logical datapath sets. In this case, thehigher-level logical datapath set serves to interconnect the lower-levellogical datapath sets, thereby interconnecting the networks.

The hierarchical controller arrangement in turn results in hierarchicalprocessing of packets by the switching elements within the network. Asstated, the higher-level network controllers generate flow tables thatimplement the highest-level logical datapath set that governs packetforwarding between end machines of the network, then pass these tablesdown to the lower-level network controllers. The lower-level networkcontrollers then incorporate the received higher-level flow entries intotheir own flow tables that implement the lower-level logical datapathsets. Each set of lowest-level network controllers generates flow tablesfor the managed switching elements within the network of the controllerset, and passes these generated flow tables to the managed switchingelements. These flow tables specify lookup entries that the managedswitching elements use to process packets within the logical network.When a switching element receives a packet on a physical port, theswitching element maps the packet to logical ports of the hierarchicaldata paths, makes a forwarding decision to an egress port at the highestlevel, and maps the identified egress port back down through thehierarchy of data paths to a physical egress port.

A. Connecting Multiple Managed Networks

The above description of FIGS. 5-9 focuses on a single managed network,with one controller (or cluster of controllers) implementing a logicalswitching element within the physical switching elements of the network.In some embodiments, the machines connected by the logical switchingelement may be physically located in a single data center or acrossseveral data centers. For instance, in FIG. 9, the rack of hosts 960might be in a first location while the racks of hosts 970 and 980 arelocated in a second location. In between two (or more locations) is aninterconnecting network of unmanaged switching elements, routers, etc.(e.g., the Internet, a different local network, etc.).

1. Single Controller Cluster

Some embodiments use a particular type of managed switching element toconnect a managed network at a particular location (e.g., a data centerwith one or more racks of host machines) to an external unmanagednetwork (e.g., in order to connect the machines at the data center tomachines at another data center through the unmanaged network). FIG. 10illustrates a network 1000 that spans two data centers 1005 and 1010.The first data center 1005 includes managed switching elements 1015,1020, and 1025. The managed switching elements 1015 and 1020 are edgeswitching elements, to which end machines (either virtual machines orphysical machines) are connected. For instance, the edge switchingelement 1015 might be a software switching element (similar to the OVSswitches in the previous section), and the edge switching element 1020might be a TOR hardware switching element.

In addition to the edge switching elements 1015 and 1020, the first datacenter 1005 includes an interconnecting managed switching element 1025,which connects to the network outside the data center 1005. Theinterconnecting managed switching element 1025, in some embodiments, isan extender. An extender is a type of managed switching elementdescribed in detail in U.S. application Ser. No. 13/177,535, filed onJul. 6, 2011 and entitled “Hierarchical Managed switching elementArchitecture”, which is incorporated herein by reference. In someembodiments, an extender enables communication between a machine withinthe managed network and external machines by adding (to incomingpackets) and removing (from outgoing packets) logical context IDs thatindicate that a packet belongs to (and is routed on) a particularlogical datapath set.

In FIG. 10, the second data center 1010 also includes several managededge switching elements 1035 and 1040 connected to several machines, aswell as an interconnecting managed switching element 1030. In someembodiments, this interconnecting managed switching element 1030 is alsoan extender, with the two extenders 1025 and 1030 communicating (throughthe external network 1045). In some embodiments, in order for these twomanaged switching elements 1025 and 1030 to communicate, a tunnel isdefined through the network 1045 between the managed switching elements.In some embodiments, the network through which the tunnel is defined isa layer 3 (L3) network, such as an Internet Protocol (IP) network. Sucha tunnel may be defined using Generic Routing Encapsulation (GRE), IPSecurity (IPSec), Stateless Transport Tunneling (STT), or othertunneling protocols. These tunnels enable packets to be transported overthe network 1045 between the data centers 1005 and 1010 with no orminimal processing by the intervening switching/routing elements thatmake up the network 1045.

In addition to extenders, some embodiments utilize pool nodes asinterconnecting managed switching elements (or use pool nodes asinternal non-edge managed switching elements within a managed network).Pool nodes are described in detail in application Ser. No. 13/177,535,which is incorporated by reference above. Pool nodes, in someembodiments, are connected to and positioned above the edge switchingelements in a hierarchical switching network architecture. In someembodiments, each edge switching element is only responsible for storingforwarding information for a subset of the machines connected by alogical datapath set. When a managed edge switching element does nothave an entry for a destination address, the edge switching elementautomatically forwards the packet to a pool node, which storesinformation for a larger subset of the logical datapath set (or for allmachines on the logical switching element). The pool node then forwardsthe packet to the appropriate next switching element in order for thepacket to reach its destination.

In the example shown in FIG. 10, all of the machines at both datacenters 1005 and 1010 are on a single logical switching element thatimplements a logical datapath set. One of ordinary skill in the art willrecognize that in some embodiments, many different logical switchingelements could be implemented at the same time by the managed switchingelements in order to connect many different sets of machines within thedata centers. These logical switching elements could connect machines inboth data centers 1005 and 1010, as shown in this figure, as well asimplementing logical datapath sets that connect machines only within oneof the data centers. For instance, VM₁, M_(P), and VM₃ might beconnected via a first logical switching element while VM₄, VM₂, M₁, andM₂ are connected via a second logical switching element.

In some embodiments, when a packet is sent from a machine (e.g., VM₁) atthe first data center 1005 to a machine (e.g., VM₃) at the second datacenter 1010, the interconnecting managed switching element 1025encapsulates the packet in a tunneling protocol. This packet istransported through the network 1045 to the interconnecting managedswitching element 1030, which removes the tunneling protocol.

To implement this logical switching element within the managed physicalswitching elements 1015-1040, a controller cluster 1050 connects to themanaged switching elements 1015-1040, and generates and passes flowentries (in the form of forwarding tables) to the switching elements. Inthis figure, as well as others within this document, traffic data paths(i.e., for transporting network data packets between end machines) areshown as solid lines, while control data paths (i.e., for managing theforwarding tables of switching elements) are shown as dashed lines.

As shown, the controller cluster 1050 connects to each of the switchingelements 1015-1035. In this figure, the controller cluster 1050 isillustrated as located outside of either of the data centers (e.g., at athird data center). However, in some embodiments, the cluster might belocated at one of the data centers or distributed across both of thedata centers 1005 and 1010. For instance, the controller cluster 1050might consist of several controller instances, some of which are locatedat the first data center 1005 and some of which are located at thesecond data center 1010. These controller instances each manage one ormore of the managed switching elements (i.e., transmit flow entries thatdefine forwarding tables to the managed switching elements). Inaddition, the controller instances communicate with each other in orderto share information (e.g., about the location of physical elementswithin the network as well as the generated logical datapath sets forimplementing a logical switching element).

2. Separate Controller Clusters

The solution for interconnecting data centers illustrated in FIG. 10(using a single logical switching element to connect all of themachines) requires that the connection between the data centers becompletely reliable. Even if the controller cluster includes instancesat both data centers, these controller instances require the ability tocommunicate with each other in order to propagate any changes to thephysical structure of the network as well as any changes to theforwarding rules. In some cases, a user enters input to defineforwarding rules into a controller instance at one of the locations, andthis information must be propagated to the controller instances at theother location. However, if connectivity to one of the data centers islost, then this could potentially cause failures even for trafficinternal to one of the data centers, as the network controllers would beunable to provide updates to the switching elements as the networkchanges (e.g., as VMs are migrated to different locations).

In addition, the users of the various machines at the data centers mightnot be the actual owners of the machines. For example, in manysituations, the data center is owned and operated by a hosting service,and numerous different customers share the use of the machines providedby the hosting services. In order to enable a customer to provision itsnetwork (e.g., to set forwarding rules), the provider may expose alogical datapath set for the customer. The customer may have its own setof machines (e.g., at its own premises) and/or machines at other hostingservices, with each such site having a separate logical datapath set forthe customer to operate. Similarly, a network carrier between thehosting services and the customer might expose its connectivity as alogical datapath abstraction as well. By doing so, the network carrierprovides a well-defined API for its customers to determine how theirpackets should be handled, without the network carrier's personnelplaying any role in these operations.

Furthermore, even within a single data center, a customer might need toseparate their machines into separate logical switching elements. Forinstance, the customer might want separate logical switching elementsfor different departments, or might simply be operating so many machinesthat it becomes infeasible or impossible to do so with a single logicalswitching element. This limit may come about based on constraints of thecontroller or due to constraints of the logical datapath service model.

Accordingly, some embodiments define multiple separate logical switchingelements implemented by separate logical datapath sets, theninterconnect these separate logical datapath sets. This subsection 2describes a decentralized solution in which the interconnection of theseparate logical datapath sets is established manually. Subsection 3describes a more centralized solution that uses multiple levels ofcontroller clusters to implement multiple levels of logical datapathsets.

FIG. 11 illustrates two data centers 1100 and 1150 with separatecontroller clusters that each manage the managed switching elementswithin their respective data centers. Within the first data center 1100,a controller cluster 1105 manages a hypervisor 1110, a pool node 1115,and an extender 1120. Similarly, within the data center 1150, acontroller cluster 1155 manages a hypervisor 1160, a pool node 1165, andan extender 1170. In the setup described above (with a single controllercluster for controlling the network), some embodiments would requirepackets being sent from a VM in data center 1100 to a VM in data center1150 to travel through the hypervisor 1110, the pool node 1115, and theextender 1120. The first VM sends a packet to the hypervisor 1110 (theedge switching element that connects to this VM), which would notrecognize the destination (in the other data center 1150) in itsforwarding tables. Thus, the hypervisor 1110 forwards the packet bydefault to the pool node 1115, which recognizes the destination addressas belonging to a machine located in the other data center and thereforeforwards the packet to the extender 1120, in order for the extender 1120to send the packet through a tunnel to the extender 1170 at the seconddata center 1150.

As shown in FIG. 11, some embodiments modify this setup in order toseparate the logical datapath set into separate logical datapath setsfor each data center while still allowing the machines at one datacenter to transmit packets to the machines at the other data center (andvice versa). In such embodiments, each data center has a separatecontroller cluster to manage the switching elements at the data center.The controller cluster 1105, for example, manages the hypervisor 1110,pool node 1115, and extender 1120, while the controller cluster 1155manages the hypervisor 1160, pool node 1165, and extender 1170. Withinthe first data center 1100, there could be numerous hypervisors and poolnodes connecting numerous end machines on a single logical datapath set(or several different sets of machines on different users' logicaldatapath sets).

In order for the machines at the first data center to communicate withmachines at a second data center, some embodiments program thecontroller clusters with a second level logical datapath set thatincludes machines from both data centers. For example, if a firstlogical datapath set includes VM_(A) and VM_(B) in a first data center,and a second logical datapath set includes VM_(C) and VM_(D) in a seconddata center, then a third logical datapath set will include all four ofVM_(A), VM_(B), VM_(C), and VM_(D) in some embodiments. This thirdlogical datapath set is sent to the controller clusters at each of thedata centers, and defines a logical switching element that connects allof the machines from the first and second logical datapath sets.

A tunnel is still created between the extenders at the two sites,although the extenders can function much like pool nodes would within asite, by enabling the transmission of packets from a machine in one areaof the logical network to a machine in another area. In addition, asshown in bold in FIG. 11, some embodiments enable a direct connectionfrom the hypervisor 1110 to the extender 1120, bypassing the pool node1115. The hypervisor 1110 (edge switching element) recognizes that apacket is destined for a machine in the other data center, andautomatically forwards the packet to the extender 1120 rather than thepool node 1115. A tunnel between the extenders 1120 and 1170 stillenables the communication between the two sites, and the details ofpacket processing and encapsulation will be described in sections below.Furthermore, the hypervisor 1110 will still forward packets destined forother machines within the data center 1100 (that do not have entrieswithin the forwarding tables of the edge switching element hypervisor1110) to the pool node 1115.

3. Multiple Levels of Controller Clusters

While FIG. 11 illustrates an architecture with separate controllerclusters at different data centers, some embodiments additionallyinvolve a second-level controller cluster that generates flow entries toconnect the separate data centers. Specifically, some embodiments defineseparate logical switching elements implemented by separate logicaldatapath sets at each data center, and interconnect these separatelogical datapath sets with a single (or more than one) second-levellogical datapath set that spans multiple data centers. In some cases,the packet processing behavior of the setup shown in FIG. 11 is the sameor similar to the packet processing behavior of the multi-levelimplementation described in the subsections below.

FIG. 12 illustrates a managed network 1200 with such a multi-level (or“federated”) implementation. As shown, the managed network 1200 alsoincludes a first data center 1205 and a second data center 1210, with anunmanaged network 1260 (in this case, an L3 network) connecting the twodata centers. As shown, the first data center includes a first levelcontroller cluster 1215, an interconnecting managed switching element1220, edge switching elements 1225 and 1230, and several end machines(both virtual and physical machines). While many of the managed networksin the following sections include both virtual and physical machines,one of ordinary skill in the art will recognize that in some embodimentsall of the machines at a data center may be virtual machines, or may allbe physical machines. The second data center 1210 also includes a firstlevel controller cluster 1235, an interconnecting managed switchingelement 1240, edge switching elements 1245 and 1250, and several endmachines.

The interconnecting managed switching elements 1220 and 1240 may bedifferent types of switching elements in different embodiments (e.g.,pool nodes, extenders). For instance, they might both be pool nodes,both be extenders, or a combination of the two types of switchingelements. These interconnecting managed switching elements create atunnel through the L3 network in order to send packets between the twonetworks when necessary. The packet processing performed by theseswitching elements in order to handle this interconnection between thedata centers will be described below.

In addition to the first level controller clusters 1215 and 1235, thefirst data center 1205 also includes a second level controller cluster1255. As discussed above, each network controller instance (or, eachlogical controller instance in the case that the cluster is separatedinto logical and physical controllers) in the first level controllercluster includes a control application and a virtualization application.The first level controller cluster generates forwarding table entries(also referred to as “flow entries”, “data flows”, or “flows”) topopulate the physical control plane of the managed switching elements.

In some embodiments, the second level controller cluster has the samestructure as the first level controller clusters, with a controlapplication and virtualization application that implement a logicalswitching element by generating flow entries. As with the first levelcontroller clusters, the second level controller cluster may be a singlenetwork controller instance or may consist of multiple controllerinstances that communicate with each other in order to update theirknowledge of the network and to disseminate information regarding thegenerated flow entries.

However, rather than connecting directly to the managed switchingelements, the second level controller cluster pushes its flow entries toone or more first level controller clusters in order for those firstlevel controller clusters to implement these second-level flows withinthe first-level flow entries that they send to the managed switchingelements. Thus, the first level controller cluster 1215 need not includeany of the machines at the data center 1210 on its logical switchingelement, as the interconnections between machines at the data center1205 and machines at the data center 1210 will be implemented by thesecond level logical switching element. As will be described in furtherdetail in Section B below, in some embodiments the virtualizationapplication of the second-level controller cluster converts flow entriesfrom the second level logical forwarding plane directly to the firstlevel logical forwarding plane. Because all of the forwarding data(e.g., port bindings, etc.) is received at the second level controllercluster, there is no control plane with regard to the forwardingdecisions. As such, the control application at the first levelcontrollers is not involved in generating flow entries for packetforwarding. However, in some embodiments, the first level logicalcontrol plane may be used to enter security or other policies (e.g., forconversion into ACL tables).

In the example of FIG. 12, the second level controller cluster 1255 islocated at one of the data centers (specifically, data center 1205). Asshown, the second level controller cluster 1255 connects to the firstlevel controller cluster 1215 at data center 1205 in order to send theflow information for the second level logical datapath sets to the firstlevel controller cluster 1215. In addition, the second level controllercluster 1255 connects to the first level controller cluster 1235,through the unmanaged network 1260. In some embodiments, the secondlevel controller cluster actually passes the control data through thetunnel between the two interconnecting managed switching elements 1220and 1240 that connects the two data centers. However, the dashed line isshown separately here in order to indicate the flow of control data (ascompared to network traffic).

There is no requirement that the second level controller must be locatedat one of the data centers, however. FIG. 13 illustrates an alternateimplementation of a managed network 1300 with multiple levels ofcontroller clusters. In this case, the second level controller cluster1305 is located outside of both of the data centers 1205 and 1210. Thislocation might be a tenant's location (i.e., the owner of the machinesinterconnected by the logical switching elements), or at a third datacenter. Some embodiments attempt to locate the second level controllerat the most well-connected of the possible locations. For instance, ifconnecting three data centers, then the second level controller might belocated at the data center with the best connection (e.g., the mostredundant and/or fastest connection) to the other two.

While the ability of the second level controller to connect to thedifferent data centers (or domains) is important, part of the goal ofthe federated approach is to ensure that the separate domains canoperate independently if a domain is cut off, either from the otherdomains or from the second level controllers. If, for example, theconnection between the first data center 1205 and the second data center1210 were to go down, each of the first level controller clusters couldstill maintain and update the logical datapath sets for their respectivesites, maintaining connectivity within the sites. Similarly, if thesecond level controller cluster 1305 were cut off, the two first levelcontroller clusters would continue managing their respective data centerlogical datapath sets.

As such, the second level controller of some embodiments does not needto be aware of the actual physical network topology within a datacenter. While the second level controller needs to account for which endmachines are located in which domain, the controller need not know wherethe different machines are located within a domain. Thus, physicalchanges such as VM migration within a data center need not be pushed upto the second level controller, as it will not affect the second levellogical switching element. This information will still be gathered bythe first level controller, as it may require updating the first levelflow entries sent to the different managed switching elements. Forinstance, if a particular VM moves from a first hypervisor to a secondhypervisor, then the first level network controller for the logicalnetwork containing the particular VM will need to generate new flowentries for various switching elements (e.g., updating pool nodes, thefirst and second hypervisors, etc.).

When an end machine is moved from one domain to a different domainconnected by the second level logical switching element, then the secondlevel controller cluster may be involved in some embodiments. In thisregard, different embodiments may distribute the functionalitydifferently between the first and second level controllers. In a morecentralized approach, the second level controller receives updates anytime a machine moves between domains, then generates and pushesforwarding table entries. In a more decentralized approach, the secondlevel controller pushes an entry indicating that the first levelcontrollers should use a distributed learning algorithm to handle themove of the end machine from one domain to the other. Such a distributedlearning algorithm might be implemented using a standard flooding-basedapproach, or by relying on a lookup service-based model that does notrequire any flooding.

The ability of the separate domains to function in isolation, however,does not mean that connectivity between the second level controllers andfirst level controllers is unimportant. As such, some embodimentsinterconnect the second level controllers to each site independentlywith enough redundant paths to provide a desired level of reliability.Some embodiments rely on gossiping to guarantee a reliable system ofupdates from the second level controller to the different first levelcontrollers at different sites. To implement this gossiping, someembodiments utilize a distributed protocol specialized for the deliveryof flow entries.

Irrespective of the approach used to disseminate updates to the flowentries, in some embodiments the second level flow entries may not makeit to switching elements at the different first level domains at exactlythe same time (e.g., because of different travel times from the secondlevel controller to the different first level controllers, or differentprocessing times at the different first level controllers). Thus, someembodiments instruct the first level controllers to include versioninginformation in the packet processing pipeline, so that the managedswitching elements add this versioning information to each packetreceived from an end machine. Thus, a single packet will use the samenetwork state version across multiple domains. In this manner, thesecond level controller can push the state in stages. Only afterpreparing a new network state version far enough, the system enables theuse of the new version at the network edge, and only after updating allof the edges to the new version will the earlier version be removed.

In addition to connecting machines on different first level logicalswitching elements at different data centers (or other locations), someembodiments use a federated approach within a single data center. FIG.14 illustrates such a data center 1400 that includes a first domainmanaged by a first level controller cluster 1405 and a second domainmanaged by a different first level controller cluster 1410. In order tointerconnect these two domains, the data center also includes a secondlevel controller cluster 1415 that implements a logical switchingelement over both of the domains, and pushes down the second level flowsto the two first level controller clusters 1405 and 1410. In this case,the interconnecting managed switching elements on the edges of thedomains are more likely to be pool nodes, as there is no externalunmanaged network through which to connect (although, as is the case inall of the figures, there may be unmanaged switching elements that arenot shown in between the managed switching elements).

Some embodiments use this federated approach within a single data centerfor different reasons. For instance, the size of a single first levellogical datapath set may be constrained, by either the underlyingcontroller implementation or by the constraints of the logical datapathservice model. While the second level logical switching element may havethe same number of logical ports as would a first level logicalswitching element connecting all the machines on a logical datapath set,the flow entries for the second level logical datapath set will besimpler because they only need to handle traffic going from one of thefirst level domains to the other first level domain.

B. Generation of Flow Entries

As mentioned above, in the federated system, flow entries are generatedinitially by the second level controller, then pushed down to theappropriate first level controller(s), which generate flow entries topush down to the managed switching elements. Within this and latersections, “1L” may be used to refer to the first level logical datapathsets and network controllers. Similarly, “2L” may be used to refer tothe second level logical datapath sets and network controllers. Theseterms are intended to be different from “L1” and “L2” that often referto the Open Systems Interconnect (“OSI”) physical and data link layers.

Section I above described the use of a network controller (or controllercluster) in a single level logical network to generate a logicaldatapath set and create flows that are pushed down to the managedswitching elements. The network controller of some embodiments includesa control application for converting logical control plane data tological forwarding plane data, and a virtualization application forconverting logical forwarding plane data to physical control plane data.In some embodiments, each of these sets of data (logical control plane,logical forwarding plane, physical control plane) are stored in thenetwork controller as nLog tables, and the control and virtualizationapplications perform nLog table mapping operations to convert from onedata plane to the next using an nLog rules engine. In some embodiments,in fact, the control application and virtualization application use thesame rules engine to perform their table mappings. The physical controlplane data is then pushed to the managed switching elements.

For a network with multiple levels of controller clusters, in someembodiments the controller clusters at each level of the hierarchyperform additional conversions of the flows (e.g., using nLog tablemapping engines). FIG. 15 conceptually illustrates a control datapipeline 1500 for a hierarchically-arranged set of network controllersat two levels that manage a federated network. Specifically, FIG. 15conceptually illustrates a 2L network controller 1505 and two separate1L network controllers 1510 and 1515 for two different 1L domains.

As shown, the 2L network controller 1505 includes a control application1520 and a virtualization application 1525. The control application 1520converts 2L logical control plane data into 2L logical forwarding planedata. In some embodiments, the control application 1520 exposesconstructs with which the control application itself or users of theapplication (i.e., the owners of the logical datapath sets) define thelogical datapath set within the logical control plane. The logicalcontrol plane data of some embodiments includes logical access controllist (“ACL”) data that may be specified by the user (e.g., to definesecurity policy). In addition, the 2L logical control plane data mayinclude logical forwarding records generated by the control applicationin response to changes in the network detected by the managed switchingelements and pushed up through the 1L network controllers. At least someof the 2L logical control data of some embodiments may be specifiedwithout consideration for a current arrangement of managed switchingelements and how the logical datapath set will actually be convertedinto data for the different switching elements.

The control application 1520 receives this 2L logical control plane dataas input, and applies its table mapping rules to generate 2L logicalforwarding plane data as output. The 2L logical forwarding plane dataincludes lookup entries that define the logical datapath set based onthe 2L logical control plane data. In some embodiments, for example, thecontrol plane might define that a particular MAC address is located at aparticular logical port. This 2L control plane data, however, does notprovide an actual lookup entry. The 2L logical forwarding plane datagenerated by the control application defines that if a packetdestination matches the particular MAC address, then the packet shouldbe forwarded to the particular logical port. This 2L control plane data,however, does not provide any context for the packet. This essentiallyturns a piece of data into a lookup entry.

The control application 1520 pushes the 2L logical forwarding plane datato the virtualization application 1525. As shown, the virtualizationapplication 1525 takes the 2L logical forwarding plane data as input,and outputs 1L logical forwarding plane data. In some embodiments, the1L logical forwarding plane data adds a match of the 2L datapath to thelookups. For instance, in the example above, the 1L logical forwardingplane lookup determines that if a packet has matched the particular 2Ldatapath (e.g., via ingress port matching) and matches the particulardestination MAC address, then the packet should be forwarded to theparticular logical port.

The virtualization application pushes the logical forwarding plane datato the two 1L network controllers 1510 and 1515. These 1L networkcontrollers, in some embodiments, receive the logical forwarding planedata customized to their particular logical domain. That is, in someembodiments, certain entries are only pushed to the network controllersthat need the entry (though this would not be the case for theattachment of a new VM for a port, which would be sent to all 1Ldomains). At each of the 1L controllers, a virtualization applicationtakes the logical forwarding plane data for the domain and converts thelogical forwarding plane data into physical control plane data for themanaged switching elements in the domain. In the above-described examplefor a newly attached machine, the physical control plane lookupdetermines that if a packet has matched the particular 1L datapath andhas matched the particular 2L datapath and matches the particulardestination MAC address, then the packet should be forwarded to theparticular logical port. In this case, two separate physical controlplane lookups are generated by the two different 1L network controllers.For the first network controller 1510, the lookup requires a match tothe particular 1L datapath defined by that network controller for its 1Ldomain. Similarly, a different lookup is generated for the networkcontroller 1515 that requires a match to its particular 1L datapath.

As with the control and virtualization applications at the 2L controller1505, the virtualization application of the 1L controller of someembodiments uses an nLog table mapping engine to perform thisconversion. The physical control plane data of some embodiments is datareadable by the managed switching elements, which the managed switchingelements convert into physical forwarding plane lookups. In someembodiments, the 1L network controllers include push the physicalcontrol plane data to the appropriate managed switching elements withinthe network.

In this figure, no logical control plane is illustrated for the 1Lnetwork controllers. Because the actual forwarding decisions are made atthe 2L level, no 1L logical control plane data is required or used insome embodiments. However, in some embodiments administrators of the 1Ldomain can use the 1L network controller to configure logical controlplane data for their particular domain. For instance, additionalsecurity policy may be set at the 1L logical control plane in someembodiments.

The above description shows a simple case in which a single 2L logicaldatapath set connects two 1L logical datapath sets. As will be describedin subsequent sections, in some embodiments additional 1L datapaths maybe connected by a 2L datapath, and additional levels of logical datapathsets may be used to provide additional discretization of the network.For a three-level network, the control data pipeline would include onlya 3L logical control plane, with the lookups converted to 3L, 2L, and 1Llogical forwarding plane data. More generally, for any number of levels,in some embodiments there is only one logical control plane, at thetopmost level, with lookups converted to logical forwarding plane dataat each subsequent level of network controller.

This example illustrates the 1L network controller converting datadirectly from the logical forwarding plane to the physical controlplane, which includes data customized to the different managed switchingelements. However, for some embodiments, this represents asimplification. In such embodiments, the 1L network controllervirtualization application converts the logical data path set from the1L logical forwarding plane to universal physical control plane (UPCP)data that is generic for any managed switching element that implementsthe logical datapath set within the 1L domain. In some embodiments, thisvirtualization application is part of a controller instance that is amaster controller for the particular logical datapath set within the 1Ldomain (also referred to as the logical controller).

In some embodiments, the UPCP data is then converted to customizedphysical control plane (CPCP) data for each particular managed switchingelement by a controller instance that is a master physical controllerinstance for the particular managed switching element, or by a chassiscontroller for the particular managed switching element. When thechassis controller generates the CPCP data, the chassis controllerobtains the UPCP data from the virtualization module of the logicalcontroller through the physical controller.

Irrespective of whether the physical controller or chassis controllergenerate the CPCP data, the CPCP data for a particular managed switchingelement needs to be propagated to the managed switching element. In someembodiments, the CPCP data is propagated through a network informationbase (NIB) data structure, which in some embodiments is anobject-oriented data structure. Several examples of using the NIB datastructure are described in U.S. patent application Ser. Nos. 13/177,529and 13/177,533, which are incorporated by reference above. As describedin these applications, the NIB data structure is also used in someembodiments to may serve as a communication medium between differentcontroller instances, and to store data regarding the logical datapathsets (e.g., logical switching elements) and/or the managed switchingelements that implement these logical datapath sets.

However, other embodiments do not use the NIB data structure topropagate CPCP data from the physical controllers or chassis controllersto the managed switching elements, to communicate between controllerinstances, and to store data regarding the logical datapath sets and/ormanaged switching elements. For instance, in some embodiments, thephysical controllers and/or chassis controllers communicate with themanaged switching elements through OpenFlow entries and updates over theconfiguration protocol. Also, in some embodiments, the controllerinstances use one or more direct communication channels (e.g., RPCcalls) to exchange data. In addition, in some embodiments, thecontroller instances (e.g., the control and virtualization modules ofthese instance) express the logical and/or physical data in terms ofrecords that are written into the relational database data structure. Insome embodiments, this relational database data structure are part ofthe input and output tables of a table mapping engine (called nLog) thatis used to implement one or more modules of the controller instances.

For a two-level network (e.g., that shown in FIG. 15), FIG. 16conceptually illustrates a process 1600 performed by the second levelnetwork controller of some embodiments to generate flow entries for anevent detected at the logical control plane. As shown, the process 1600begins by receiving (at 1605) an update to the 2L logical control plane.Such an update may be a user entering a particular ACL policy (e.g.,enabling port security or machine isolation for a particular port,requiring a particular QoS for a particular machine at a particularport, etc.). In addition, updates may be received at the control planeafter being pushed upwards from the managed switching elements. When anew machine is attached to a particular managed switching element, thisnetwork information is pushed up to the 1L network controller thatmanages the particular switching element, then from the 1L controller tothe 2L controller. In some embodiments, the user will have configuredthe logical ports (of the first and second logical switching elements)to which the newly detected machine should be bound.

The process then determines (at 1610) whether the update to the 2Llogical control plane requires the creation of new flow entries. Forinstance, if the update simply indicates that the network has notchanged, then no new entries will be required, and the process ends. Onthe other hand, if the update specifies new ACL rules, or indicates theattachment of a new VM, then the 2L network controller will begingenerating new flow entries.

Next, the process 1600 translates (at 1615) the 2L logical control planeupdate into an update to the lookups in the 2L logical forwarding plane.As mentioned above, in some embodiments this translation involvesturning a piece of data into a lookup entry. Throughout the process 1600and the process 1700 (shown in FIG. 17, described below), severalexamples will be used to illustrate the translation of logical controlplane data into physical control plane data. The first example is thatgiven in simple terms above: the attachment of a new machine, having aMAC address A, to a particular logical port X of the 2L logical datapathset. In this case, the logical control plane data states “MAC A is atlogical Port X”. The control application translates this event into anupdate to the 2L LDPS lookup table that reads “If destination matchesMAC A, forward to Port X”.

The second example described here illustrates the generation of ACLtable entries. While the conversions related to the hierarchical networkare feature agnostic (e.g., the addition of match conditions), exampleswill be given for the specific example of port security. Thus, at thelogical control plane, the user specifies that a particular port shouldbe secured (e.g., Port X)—that is, that network data entering andexiting the logical switching element through the particular port haveonly certain addresses that the switching element has restricted theport to use. For this example, the user has secured Port X, restrictedto MAC address A and IP address B. Other examples of ACL rules that auser could specify include counters (i.e., counting the number ofpackets coming from a particular source address or to a destinationaddress), machine isolation (i.e., only sending broadcast/multicastpackets received from a particular machine to a particular set ofmachines), QoS enablement (i.e., requiring a particular quality ofservice for packets sent from or to a particular port), etc.

For the port security, the control plane data simply specifies that PortX should be secure, with MAC A and IP B as the allowed addresses. Thelogical forwarding plane converts this into an ingress ACL entry and anegress ACL entry. At this point, the entries are similar to those thatwould be used in a single level logical network. For the ingress ACL,the entry specifies that a packet entering the logical switching elementfrom Port X is allowed if the packet has “A” as the source MAC addressand “B” as the source IP address, and dropped if the MAC address or IPaddress are different. Furthermore, Address Resolution Protocol (“ARP”)responses must correspond to the correct address (i.e., the MAC addressmust be “A” and the IP address must be “B” in the source response, orthe packet will be dropped). Logically, the lookup instruction mightstate “If received from Ingress Port X→Allow, or Drop If ARP MAC not Aor IP not B, or Drop If MAC not A or IP not B”. This effectivelyprevents the machine at Port X from using other MAC or IP addresses thanthose assigned to it. Similarly the egress ACL lookup prevents packetsnot sent to the correct address from exiting the switching element atPort X, with an instruction of “If sent to Egress Port X→Drop If dest.IP not B”. This prevents other IP addresses from being used at Port X;by the nature of the logical forwarding, packets sent to MAC addressesother than A will not be directed to Port X in the first place.

After translating the logical control plane update into a 2L logicalforwarding plane lookup(s), the process then translates (at 1620) the 2Llogical forwarding plane data into 1L logical forwarding plane lookupentries. As with the 2L logical control plane to 2L forwarding planetranslation, in some embodiments the conversion from 2L logicalforwarding plane to 1L logical forwarding plane is performed as an nLogtable mapping operation (e.g., using the same table mapping engine asfor the 2L logical control plane to 2L logical forwarding operation).For both the ACL lookups and the attachment of a new machine, thevirtualization application adds a match of the 2L LDPS to the entry.Thus, the first entry now states “If match 2L LDPS and destinationmatches MAC A, forward to Port X”. The ingress ACL entry at the 1Llogical forwarding plane reads “If match 2L LDPS and If received fromIngress Port X→Allow, or Drop If ARP MAC not A or IP not B, or Drop IfMAC not A or IP not B”. Similarly, the egress ACL entry reads “If match2L LDPS and If sent to Egress Port X→Drop If dest. IP not B”.

In addition to translating the 2L logical forwarding lookups to 1Llogical forwarding lookups, the process 1600 also generates (at 1625)additional 1L logical forwarding plane entries in order to realize the2L lookups within the 1L logical forwarding plane. In some embodiments,the virtualization application rules engine creates additional flowentries to handle the operations around the forwarding lookups. Theseinclude ingress and egress port integration lookups to handle theingress context mapping and egress context mapping described below, aswell as additional entries to handle the tunnels between different 1Ldomains (i.e., for packets not originating in the destination domain).

In some embodiments, these lookup entries are generated as soon as a newmachine is added at a particular port and the 1L and 2L logical portsare bound to the same machine. Thus, when ACL entries are generated fora particular port, these additional 1L logical forwarding entries arenot affected. Continuing to refer to the example of a machine with MACaddress A and IP address B at Port X, the virtualization application ofsome embodiments generates four types of entries: ingress portintegration, egress port integration, tunnel sending, and tunnelreceiving. The ingress port integration entry matches the 1L port (“PortK”) to which the machine is bound to the 2L Port X to which the machineis also bound. Thus, this entry states “If received from 1L ingress PortK→Mark 2L ingress as Port X”. In order to generate such an entry, thevirtualization application uses input tables that correspond the portbindings between the two levels (based on user-provided information).

The egress port integration entry matches a forwarding decision at the2L level to a 1L port. Specifically, for the continuing example, theegress port integration entry states “If sent to Port X→Run throughegress pipeline then send to Port K”. Thus, a packet forwarded to Port Xat the 2L level will be first sent to the egress pipeline (at whichpoint the egress ACL rules are applied to the packet), then sent to the1L Port K. As with the ingress port integration, input tables specifyingport bindings are used by the virtualization application to generatesuch egress port integration entries. In some embodiments, both theingress and egress port integration lookups are written specifically forthe 1L domain that actually contains Port K. Only packets originatingfrom Port K will need to be mapped to Port X, and packets destined forPort X will only need to be mapped to Port K once received at the 1Ldomain.

In addition to the ingress and egress port integration lookups, someembodiments also generate lookups to handle the receipt and transmissionof packets sent through different 1L LDPS than the 1L LDPS containingPort X. The ingress 2L port and the egress 2L port might be in different1L LDPS, and therefore the packet will have to travel along the tunnelsbetween these 1L domains. Thus, two lookups (at the sending side of thetunnel and the receiving side of the tunnel) are generated for the 1Llogical forwarding plane. For any 1L LDPS that does not contain Port Xand that connects directly to the 1L LDPS that does contain Port X, thevirtualization application generates a tunnel encapsulation lookup entrythat states “If sent to Port X→Encapsulate with X's context ID andoutput to 1L Port that connects to destination 1L LDPS”. For thereceiving side of the tunnel (i.e., the 1L LDPS that contains Port X),the virtualization application generates a tunnel decapsulation lookupentry that states “If tunneled→Decapsulate to identify 2L port, thenResubmit”. The resubmission results in the execution of the egress portintegration described above.

In some embodiments, the 1L datapaths are not a full mesh. That is,there may not be a direct connection between each pair of 1L domainswithin the 2L domain. In some such embodiments, additional lookupentries are generated for sending packets to the appropriate tunnelsfrom 1L domains that do not directly connect to the domain containingPort K. For each such 1L domain, the virtualization applicationgenerates a lookup stating that “If sent to Port X→Encapsulate with X'scontext ID and output to 1L Port that connects to appropriate next 1LLDPS”. Correspondingly, for any 1L domain that does not contain Port Kbut that can receive a packet destined for MAC B over such a tunnel, thevirtualization application generates a lookup stating that “Iftunneled→Decapsulate to identify 2L port, then Resubmit”. Theresubmission will then result in the eventual execution of the nexttunnel to the next 1L datapath.

To generate the tunneling lookups, the rules engine of thevirtualization application uses an input table that defines theinterconnections between the different 1L logical datapath sets. Such aninput table defines the ports of the 1L LDPS that connect to other 1LLDPS. In addition, some embodiments use tables generated at the 2Lcontrol plane that define pathways through the 1L datapaths for packetsthat originate in a 1L domain that does not directly connect to thedestination 1L domain (i.e., whether packets should be forwarded througha first domain or a second domain, that both directly connect to thedestination 1L domain). In different embodiments, these pathways may bedefined by a user or by an optimization algorithm that combines networkdata with user-entered traffic policies (e.g., QoS guarantees, trafficclassification, etc.).

With all of the 1L logical forwarding plane entries generated for theupdate, the process 1600 identifies (at 1630) the 1L controllers toreceive the generated lookups. As described for some of the differentlookup entries above, not all controllers will receive every lookup, assome lookups may not be needed for the managed switching elements insome 1L domains. For example, the tunnel sending lookups will not needto be sent to the 1L domain containing the destination port, and thetunnel receiving lookup will not need to be sent to the 1L domains thatwill never receive packets sent to the destination port. Depending onthe complexity of the network and the defined pathways through thedomains, in some cases all domains might be potential recipients. Theegress and ingress port integration lookups, as well as certain egresspipeline entries (e.g., the egress ACL for port security) will only besent to the 1L controller at the 1L domain containing the port beingmapped. On the other hand, the forwarding entries and the ingresspipeline entries of some embodiments are sent to all of the 1L domains,as any of these domains might have a machine sending a packet to themachine at Port X.

Finally, the process 1600 pushes (at 1635) the generated flow entries tothe identified 1L controllers, then ends. In some embodiments, the 2Lcontroller propagates the generated flow entries to the 1L controllerthrough an object-oriented (NIB) data structure, while other embodimentsuse direct communication channels (e.g., RPC calls) to exchange the flowentries.

FIG. 17 conceptually illustrates a process 1700 of some embodiments thatgenerates the physical control plane data from the 1L forwarding planedata received at the 1L controller from the 2L controller. In someembodiments, a virtualization application at the 1L controller performsthe process 1700. As shown, the process begins by receiving (at 1705) 1Llogical forwarding plane lookup entries from the 2L controller. In someembodiments, the virtualization application receives these lookupentries, published by the 2L controller, by using its subscriber tables.In some embodiments, the subscriber tables specify locations of 2Lcontrollers from which the virtualization application should retrieveflow entries. As described above, the only logical control plane forthese flows exists at the 2L logical level. In some embodiments,however, local policies for traffic contained within a particular 1Ldomain may be set at the control plane of the 1L logical controller.

Next, the process translates (at 1710) the 1L logical forwarding planelookups into physical control plane lookups. As with the translationoperations performed by process 1600, in some embodiments the conversionfrom 1L logical forwarding plane to physical control plane is performedas an nLog table mapping operation. The rules engine used by the 1Lcontroller is the same rules engine (with different input tables) asthat used by the 2L controller in some embodiments. For both the ACLlookups and the forwarding entry for the new machine attached to thenetwork, the virtualization application adds a match of the 1L LDPS tothe entry. Thus, the entry for forwarding packets to Port X now states“If match 1L LDPS and If match 2L LDPS and If destination matches MACA→forward to Port X”. The ingress ACL entry at the physical controlplane reads “If match 1L LDPS and If match 2L LDPS and If received fromIngress Port X→Allow, or Drop If ARP MAC not A or IP not B, or Drop IfMAC not A or IP not B”. Similarly, the egress ACL entry reads “If match1L LDPS and If match 2L LDPS and If sent to Egress Port X→Drop If dest.IP not B”. This egress ACL entry, in some embodiments, is only generatedby the 1L network controller at the domain containing Port X, as theother 1L network controllers do not receive the entry at their logicalforwarding planes.

In addition to translating the 1L logical forwarding plane lookups tophysical control plane lookups, the process 1700 also generates (at1715) additional physical control plane lookup entries to realize the 1Llogical forwarding plane over the physical network. In some embodiments,the 1L virtualization application rules engine creates additional flowentries to handle the operations around the forwarding lookups. Theselookups are the analogues to the entries generated by the 2Lvirtualization application for realizing the 2L logical forwarding planeon top of the 1L logical forwarding plane. As with the entries describedearlier, the lookups generated at 1715 include ingress and egress portintegration entries to handle ingress context mapping and egress contextmapping, as well as additional entries to handle the tunnels betweendifferent managed switching elements.

In some embodiments, these lookup entries are generated as soon as a newmachine is added at a particular physical port (e.g., a virtualinterface) and the 1L and 2L logical ports are bound to that physicalport. When ACL entries are generated for a particular port, theseadditional physical control plane entries are not affected. For the sakeof the examples, the physical port to which the machine located at 1Llogical Port K and 2L logical Port X is connected is Port J in theongoing examples. Continuing with this example, the ingress portintegration entry for this level matches the physical Port J to whichthe machine is connected to the 1L port K to which the machine is bound.Thus, this entry states “If received from physical ingress Port J→Mark1L ingress as Port K”. In order to generate such an entry, thevirtualization application uses input tables that correspond the portbindings between the physical and logical levels (based on user-providedinformation).

The egress port integration entry matches a forwarding decision, asmapped to the 1L logical level, to a physical port. Specifically, forthe continuing example, the egress port integration entry becomes “Ifsent to Port K→Run through egress pipeline then send to Port J”. Thus, apacket forwarded to Port K at the 1L level will be first sent to theegress pipeline and then sent to the physical Port J. As with theingress port integration, input tables specifying port bindings are usedby the 1L virtualization application to generate such egress portintegration entries. In some embodiments, both the ingress and egressport integration lookups are written specifically for the managedswitching element (e.g., a hypervisor) that actually contains Port J.Only packets originating at this port will need to be mapped to Port K,and packets destined for Port K will only need to be mapped to physicalPort J once received at the managed switching element.

In addition to the ingress and egress port integration lookups, someembodiments also generate lookups to handle the receipt and transmissionof packets through different managed switching elements within the 1Ldomain containing Port K. Packets might originate at other managedswitching elements within the domain, or originate in a different 1Ldomain and enter via an interconnecting switching element. In this case,the packet will have to travel along at least one tunnel between twomanaged switching elements within the 1L domain in order to reach theswitching element containing Port J. Thus, two lookups (at the sendingand receiving sides of the tunnel) are generated for the physicalcontrol plane. For any managed switching element that does not containPort J and that connects directly to the switching element that doescontain Port J (the interconnecting managed switching element for the 1Ldomain is often such a switching element), the virtualizationapplication generates a tunnel encapsulation lookup entry that states“If sent to Port K→Encapsulate with K's context ID and output tophysical port via tunnel that connects to destination switch”. For thereceiving side of the tunnel (i.e., at the managed switching elementthat contains Port J), the virtualization application generates a tunneldecapsulation lookup entry that states “If tunneled→Decapsulate toidentify 1L port, then Resubmit”. The resubmission results in theexecution of the egress port integration described above.

In some embodiments, the managed switching elements may not be a fullmesh. Packets incoming from other 1L domains are generally sent directlyfrom the interconnecting managed switching element to the destinationmanaged switching element in some embodiments. On the other hand, fornetworks with pool nodes, the path from a first managed edge switchingelement within the 1L domain passes through one or more pool nodes inorder to reach a second managed edge switching element (to which thedestination machine connects) in some cases. In some such embodiments,additional lookup entries are generated for sending packets to theappropriate tunnels from managed switching elements that do not directlyconnect to the destination managed switching element containing Port J.For each such switching element, the virtualization applicationgenerates a lookup stating that “If sent to Port K→Encapsulate with K'scontext ID and output to physical port via tunnel that connects toappropriate next managed switch”. Correspondingly, for any managedswitching element that does not contain Port J but can receive a packetdestined for MAC B over such a tunnel, the virtualization applicationgenerates a lookup stating that “If tunneled→Decapsulate to identify 1Lport, then Resubmit”. In this case, the resubmission results in theeventual execution of the next tunnel to the next managed switchingelement.

To generate the tunneling lookups, the rules engine of thevirtualization application uses an input table that lists the physicalports connected to each managed switching element, as well as an inputtable defining the connections between the different managed switchingelements (i.e., indicating through which additional switching elements apacket will have to travel in order to reach a particular destinationswitching element from a particular originating switching element). Inaddition, to configure the tunnels, some embodiments use an input tablethat identifies which types of tunnels are allowed for particularmanaged switching elements (e.g., hypervisors, pool nodes, extenders,etc.). In order to connect two managed switching elements, the rulesengine of some embodiments selects a shared tunnel type.

With all of the physical control plane entries generated for the updatereceived at the logical forwarding plane, the process 1700 identifies(at 1720) the managed switching elements to receive the generatedlookups. As described for some of the various lookup entries generatedat 1710 and 1715, not all of the managed switching elements within the1L domain will receive every lookup. For example, the tunnel sendinglookups will not be sent to the managed switching element to which thedestination machine actually connects, and the tunnel receiving lookupwill not need to be sent to any managed switching elements that willnever receive packets sent to the destination machine in question.Depending on the complexity of the pathways through the 1L domain, insome cases all of the switching elements might be potential recipients.In many situations though, at least some of the other edge switchingelements will never receive packets destined for the machine in question(except directly from the machines connected to those edge switchingelements). The egress and ingress port integration lookups, as well ascertain egress pipeline entries (e.g., the egress ACL for port security)will only be sent to the managed edge switching element containing thesecured port. On the other hand, the forwarding entries and ingresspipeline entries of some embodiments are sent to all of the managedswitching elements within the domain, as any of these switching elementsmight have a machine sending a packet to the machine at Port J.

Finally, the process 1700 pushes (at 1725) the generated flow entries tothe identified managed switching elements, then ends. In someembodiments, the 1L network controller communicates directly with themanaged switching elements. However, in other embodiments, the 1Lnetwork controller that performs the conversion of the 1L logicalforwarding plane data into the physical control plane data sendsphysical control plane data to master controllers for the particularswitching elements that are to receive the data, and these mastercontrollers push the data to the switching elements. In addition, whilethis example describes the computation of physical control plane datacustomized for particular switching elements (e.g., with port numbers ofthe particular switching elements), some embodiments compute universalphysical control plane data that is generic to any particular switchingelement. In this case, either the master controller or a chassiscontroller at the managed switching element performs the conversion tocustomized physical control plane data for the managed switchingelements. In some embodiments, the 1L controller propagates thegenerated flow entries (e.g., to the master controller, from the mastercontroller to the managed switching elements) through an object-oriented(NIB) data structure, while other embodiments use direct communicationchannels (e.g., RPC calls, OpenFlow entries, updates over theconfiguration protocol) to exchange the flow entries.

FIG. 18 conceptually illustrates some of these input and output tablesthrough the various flow generation operations of some embodiments.Specifically, FIG. 18 conceptually illustrates the input and outputtables for a 2L control application 1805, a 2L virtualizationapplication 1810, and one of the 1L virtualization applications 1815.The 2L control application 1805 and 2L virtualization application 1810are located in the same network controller, while the 1L virtualizationapplication 1805 is located in one of several different 1L networkcontrollers for several different 1L domains (other controllers notshown).

As shown, the control application 1805 includes an API 1820, inputtables 1825, a rules engine 1830, output tables 1835, and a publisher1840. The API 1820 provides an interface for translating input into thecontrol plane input tables 1825. This API 1820 may be used by varioustypes of management tools with which a user can view/and or modify thestate of a multi-level logical network (in this case, a two levelnetwork). In some embodiments, the management tools provide a userinterface such as a graphical user interface that allows a visualconfiguration of port bindings, ACL rules, etc. (e.g., through a webbrowser). Alternatively, or in conjunction with the graphical userinterface, some embodiments provide the user with a command line tool orother type of user interface.

Based on the information received through the API, as well as updates tothe network state received from the 1L controller (not shown), thecontrol application generates the input tables 1825. The input tablesrepresent the state of the logical switching elements managed by theuser in some embodiments. As shown in this figure, some of the inputtables include the association of MAC addresses/IP addresses withlogical ports of the 2L logical switching element, as well as ACL rulesset by the user. In this case, the Port X is associated with MAC addressA and IP address B, and is secured.

The rules engine 1830 of some embodiments performs various combinationsof database operations on different sets of input tables 1825 topopulate and/or modify different sets of output tables 1835. Asdescribed in further detail in U.S. application Ser. No. 13/288,908,incorporated herein by reference, in some embodiments the rules engineis an nLog table mapping engine that maps a first set of nLog tablesinto a second set of nLog tables. The output tables 1835 populated bythe rules engine of the control application 1805 include 2L logicalforwarding plane lookups (e.g., mapping a MAC address to a destinationoutput port) and 2L logical forwarding plane ACL entries (e.g., securingPort X).

The publisher 1840 is also described in further detail in U.S.application Ser. No. 13/288,908, and publishes or sends the outputtables 1835 to the virtualization application 1810, in order for thevirtualization application to use the output tables 1835 among its inputtables. In some embodiments, the publisher 1840 also outputs the tablesto an object-oriented data structure (NIB) that stores network stateinformation.

The 2L virtualization application 1810 receives the output tables 1835of the control application 1805, and converts this 2L logical forwardingplane data to 1L logical forwarding plane data. As shown, the 2Lvirtualization application 1810 includes a subscriber 1845, input tables1850, a rules engine 1855, output tables 1860, and a publisher 1865. Thesubscriber 1845 of some embodiments is responsible for retrieving tablespublished by the publisher 1840 of the control application 1805. In someembodiments, the subscriber 1845 retrieves these tables from the sameobject-oriented data structure to which the publisher stores the tableinformation. In other embodiments, a change in the tables is detected bythe virtualization application in order to initiate the processing.

The input tables 1850 include, in some embodiments, at least some of theoutput tables 1835, in addition to other tables. As shown, in additionto the 2L logical forwarding plane data, the input tables 1850 includeport binding information that indicates, for each 2L port, the 1L portbound to the same MAC and IP address (and that 1L port's 1L logicaldatapath set). In addition, some embodiments include interconnectioninformation that describes the pathways for packets to take through thedifferent 1L domains. In some embodiments, this information is generatedby the user through the user interface. Other embodiments use userpolicies and network data to optimize the pathways, as described belowby reference to FIG. 46.

In some embodiments, the rules engine 1855 is the same as the rulesengine 1830. That is, the control application 1805 and thevirtualization application 1810 actually use the same rules engine insome embodiments. As indicated, the rules engine performs variouscombinations of database operations on different sets of input tables1850 to populate and/or modify different sets of output tables 1860. Insome embodiments, the rules engine is an nLog table mapping engine thatmaps a first set of nLog tables into a second set of nLog tables. Theoutput tables 1860 populated by the rules engine 1855 include 1L logicalforwarding plane lookups (e.g., mapping a MAC Address to a destinationlogical port when the 2L LDPS is matched) and 1L logical forwardingplane ACL entries (e.g., securing Port X). In addition, the ingress andegress port integration and tunnel sending/receiving lookups aregenerated by the rules engine 1855 in some embodiments. In addition tothe information shown in the figure, some embodiments also include inthe output tables the correct 1L network controllers to receive thedifferent tables.

The publisher 1865 is similar to the publisher 1840 in some embodiments.The publisher 1865 publishes and/or sends the output tables 1860 to the1L network controllers, including the controller containingvirtualization application 1815. In some embodiments, the publisher 1865also outputs the tables to an object-oriented data structure (NIB) thatstores network state information.

The 1L virtualization application 1815 is located at one of the 1Lnetwork controllers in the two level federated network. Specifically, inthis case, the virtualization application 1815 is part of the networkcontroller located in the 1L domain at which the machine with MACaddress A (located at Port X of the 2L LDPS) is also located. As such,the 1L virtualization application generates flows that include those forthe managed switching element to which this machine directly connects.

As shown, the 1L virtualization application 1815 includes a subscriber1870, input tables 1875, a rules engine 1880, output tables 1885, and apublisher 1890. The subscriber 1870 of some embodiments is responsiblefor retrieving tables published by the publisher 1865 of the 2Lvirtualization application 1810 (specifically, the tables for itsparticular 1L controller). In some embodiments, the subscriber 1870retrieves these tables from the same object-oriented database to whichthe publisher stores the table information. In other embodiments,changes to the output tables from the 2L virtualization application aretransmitted via RPC calls to the 1L network controller.

The input tables 1875 include, in some embodiments, at least some of theoutput tables 1860, in addition to other tables. As shown, in additionto the 1L logical forwarding plane data generated by the 2Lvirtualization application 1810 that is appropriate to the particular 1Lnetwork controller, the input tables 1875 include additional portbinding information (matching 1L logical ports with physical ports ofparticular managed switching elements). In addition, some embodimentsinclude interconnection information that describes pathways through themanaged switching elements of the 1L network for packets either fullycontained within the network (as shown in the example pathway) orexiting/entering the network to/from other 1L domains. In someembodiments, this information is generated by the user at the 2L networkcontroller and passed down to the 1L network controller, or may begenerated at the 1L network controller.

In some embodiments, the rules engine 1880 is similar to the rulesengines 1855 and 1830. In the situation in which the 1L networkcontroller is located in the same hardware as the 2L network controller(i.e., the same physical machine), the 1L virtualization application1815 may use the same rules engine as the 2L network controller. Asindicated, the rules engine performs various combinations of databaseoperations on different sets of input tables 1875 to populate and/ormodify different sets of output tables 1885. In some embodiments, therules engine is an nLog table mapping engine that maps a first set ofnLog tables into a second set of nLog tables. The output tables 1885populated by the rules engine 1880 include physical control planelookups (e.g., mapping a MAC Address to a destination logical port whenthe 1L and 2L LDPS is matched) and physical control plane ACL entries(e.g., securing Port X). In addition, the ingress and egress portintegration and tunnel sending/receiving lookups are generated by therules engine 1880 in some embodiments. In addition to the informationshown in the figure, some embodiments also include in the output tablesthe correct managed switching elements to receive the different tables.

Finally, the publisher 1890 is similar to the publisher 1865 in someembodiments. The publisher 1890 publishes and/or sends the output tables1885 to the managed switching elements within the domain of the 1Lnetwork controller containing virtualization application 1815. Thesemanaged switching elements may include hypervisors, pool nodes,extenders, etc. In some embodiments, the publisher 1890 outputs thetables to an object-oriented data structure (NIB) that stores networkstate information.

One of ordinary skill in the art will recognize that the input andoutput tables shown in this figure are simplified conceptualrepresentations of the actual tables, which are generated in a databaselanguage appropriate for the rules engine (e.g., nLog) and may provideadditional information to that shown. Furthermore, different embodimentswill use different sets of tables. For instance, the port binding tablesof some embodiments are actually a single table that binds a particularMAC address and IP address at a particular physical port of a particularhypervisor to particular 1L and 2L logical ports.

C. Packet Processing

The above section described the generation of the forwarding tableentries for a federated network, through the second level and firstlevel network controllers. The following section describes theprocessing of packets being sent from one machine to another in such afederated network, with the managed switching elements using the flowsgenerated as described above.

FIG. 19 illustrates a set of logical datapath sets (or logical switchingelements) for an example federated network of some embodiments, whichwill be used in some of the examples below. Specifically, FIG. 19illustrates an originating first level logical datapath set 1905 (LDPSK), a destination first level logical datapath set 1910 (LDPS Z), and aconnecting second level logical datapath set 1915.

As shown, the first level LDPS K 1905 has four ports for the VMs A, B,C, and D, as well as a fifth port that connects to the LDPS Z. The firstlevel LDPS Z 1910 also has four ports for the four VMs E, F, G, and H,as well as a fifth port that connects to the LDPS K. The first levelLDPS K 1905 logically connects machines within a first domain (e.g., afirst data center) while the first level LDPS Z 1910 logically connectsmachines within a second domain (e.g., a second data center). The secondlevel LDPS 1915 that logically connects these two domains has eightports, for the eight VMs A, B, C, D, E, F, G, and H.

The arrows in this figure illustrate logical mappings for a packet sentfrom VM A in the first domain to VM H in the second domain. As will bedescribed, these logical mappings are performed at one or more managedswitching elements using the forwarding tables generated as described inthe above section. In this case, the packet enters the logical switchingelement 1905 at Port 1, and is mapped to Port 1 of the second levellogical switching element 1915. The forwarding tables that realize thesecond level logical switching element recognize the destination addressof the packet as VM H and forward the packet to Port 8 of the secondlevel logical switching element 1915 (actually exiting the logicalswitching element 1905 through Port 5). Port 8 of the second levellogical switching element then maps to Port 4 of the logical switchingelement 1910, the first level logical switching element for the second(destination) domain (actually entering the logical switching element1910 through Port 5). These mappings will be described in detail byreference to FIGS. 20-25.

FIG. 20 conceptually illustrates the path of a packet 2000 through fourmanaged switching elements between its source machine in a first domainand its destination machine in a second domain. The operation of some ofthe managed switching elements shown in this figure will be described inpart by reference to FIGS. 21, 22, and 23, which conceptually illustrateprocesses performed by some of the managed switching elements in afederated network in order to process and forward packets.

As shown, the packet 2000 originates from a source machine with apayload 2005 and headers 2010. The payload 2005 contains the actual dataintended for the destination machine, while the headers 2010 includeinformation appended by the source machine in order to enable the packet2000 to reach the destination machine. For instance, the headers 2010include the source and destination machines' addresses (e.g., MACaddresses, IP addresses, etc.). These addresses are physical addressesfrom the perspective of the machines, which are not aware of the logicalnetwork. From the perspective of the switching elements within thenetwork between the machines, however, these MAC addresses areconsidered logical addresses, as the initial managed switching elementuses the destination address to perform logical forwarding, and theaddress is kept within the encapsulation and therefore not visible tothe subsequent logical switching elements.

Thus, the packet leaves the source machine without any sort of logicalcontext ID. Instead, as described further below, all of the logicalcontext information is added and removed at the managed switchingelements. The end machines, and the network interfaces of the endmachines, need not be aware of the logical network over which the packetis sent. As a result, the end machines and their network interfaces donot need to be configured to adapt to the logical network. Instead, onlythe managed switching elements are configured by the networkcontrollers.

The packet 2000 is first sent to a source hypervisor 2015 in thisexample. In this case, the source machine from which the packetoriginates is a virtual machine (e.g., VM A of FIG. 19) that operates ona hypervisor running on a physical machine. Such hypervisors, in someembodiments, also contain software to operate as managed switchingelements, performing physical and logical forwarding functions forpackets originating from and destined for the virtual machines operatingon the hypervisor. As shown, the source hypervisor 2015 executes a firstlevel flow, which includes performing second level functions. As such,the source hypervisor 2015 executes the 2L flow to add 2L egressinformation to the packet, and then completes the 1L flow to add 1Legress information to the packet. The packet 2000 exits the sourcehypervisor with two layers of encapsulation: 2L egress information 2020encapsulated inside 1L egress information 2025.

FIG. 21 conceptually illustrates in greater detail a process 2100 ofsome embodiments for processing packets by a first hop managed switchingelement (e.g., the source hypervisor 2015) in a federated network. Thefirst hop managed switching element is the first managed switchingelement at which a packet arrives after being sent by its sourcemachine. In the case of a packet from a virtual machine, this first hopmanaged switching element is often the hypervisor on which the virtualmachine operates. As in the example of FIG. 20, in some embodiments allof the second level processing, as well as the originating LDPS's firstlevel processing, is performed at the first hop.

As shown, the process 2100 begins (at 2105) by receiving a packet from alocal machine at a physical ingress port. A switching element, whethermanaged or unmanaged, has several physical ports through which packetsmay enter or exit. In general, each port can serve as both an ingressport (for incoming packets) and an egress port (for outgoing packets),although in some embodiments certain ports may be reserved for eitheringress or egress specifically. The packet is received through aphysical port of the switching element to which the source machine ofthe packet connects, either directly or through other (unmanaged)switching elements. In the case of a packet sent from a virtual machine,this switching element is often running on the hypervisor on which thevirtual machine operates.

Based on the physical ingress port of the packet, the process determines(at 2110) the local first level logical ingress port for the packet.That is, the managed switching element maps the physical ingress port toa 1L logical ingress port. In some embodiments, the managed switchingelement bases this mapping solely on the physical port through which thepacket arrived, and the fact that the packet is not yet encapsulatedwith any 1L information. In other embodiments, the managed switchingelement also uses the packet headers (e.g., the MAC address of thesource machine) to determine the mapping.

Next, the process determines (at 2115) the second level (2L) logicalingress port for the packet based on the first level logical ingressport. In some embodiments, each 1L logical port corresponds to a secondlevel logical port. For example, in FIG. 19, Port 1 of the 1L LDPS K1905 corresponds to Port 1 of the 2L LDPS 1915. As shown in FIG. 21,operation 2115 is the beginning of the second level processing, which isencompassed within the first level processing (i.e., the first levellogical forwarding tables realize the second level logical processingpipeline).

Now that the forwarding tables have begun 2L processing, the processdetermines (at 2120) the 2L logical egress port for the packet using thelogical forwarding tables and the packet destination. That is, themanaged switching element examines the packet headers to determine thepacket destination (e.g., the MAC address), and maps the identifieddestination to a logical port of the 2L logical switching element. Thislogical forwarding operation may be performed for layer 2 processing(e.g., using the MAC address), layer 3 processing (e.g., using anInternet Protocol (IP) address), or using any other type of networkaddresses. That is, the concept of having hierarchical networkcontrollers creating hierarchical logical datapath sets is notrestricted to any particular type of network forwarding.

In addition to making a forwarding decision (i.e., mapping to a 2Llogical egress port), some embodiments also perform other forwardingtable operations within the 2L processing. For instance, someembodiments perform ingress and/or egress Access Control List (“ACL”)lookups that may contain instructions to drop a packet (e.g., if thesource of the packet is known to be corrupted), queue a packet (e.g., toenforce quality of service controls), allow a packet through, etc.

After determining the 2L logical egress port, the process encapsulates(at 2125) the packet with this second level logical egress portinformation. That is, the managed switching element prepends informationto the packet (e.g., a logical context) that includes the 2L egress portinformation. An example of such a logical context for OSI Layer 2processing is described in detail in U.S. application Ser. No.13/177,535, incorporated by reference above. The logical contextdescribed therein is a 64-bit tag that includes a 32-bit virtual routingfunction field (for representing the logical switching element to whichthe packet belongs (i.e., the 2L logical switching element)), a 16-bitlogical inport field (i.e., the ingress port on the 2L switchingelement), and a 16-bit logical outport field (i.e., the 2L egress port).

Some embodiments, however, only include the logical egress port withinthe logical context prepended to the packet. That is, the logicalcontext that encapsulates the packet does not include an explicit tenantID. Instead, the logical context captures the logical forwardingdecision made at the first hop. From this, the LDPS ID (i.e., the LDPSto which the packet belongs) can be determined implicitly at laterswitching elements by examining the logical egress port (as that logicalegress port belongs to a particular logical switching element). Thisresults in a flat context identifier, meaning that the switching elementdoes not have to slice the context ID to determine multiple pieces ofinformation within the ID. In some embodiments, the egress port is a32-bit ID. However, the use of software switching elements as themanaged switching elements that process the logical contexts in someembodiments enables the system to be modified at any time to change thesize of the logical context (e.g., to 64 bits or more), whereas hardwareswitching elements tend to be more constrained to using a particularnumber of bits for a context identifier. In addition, using a logicalcontext identifier such as those described herein results in an explicitseparation between logical data (i.e., the egress context ID) andsource/destination address data (i.e., MAC addresses). While the sourceand destination addresses are mapped to the logical ingress and egressports, the information is stored separately within the packet.

Such logical networks, that use encapsulation to provide an explicitseparation of physical and logical addresses, provide significantadvantages over other approaches to network virtualization, such asVLANs. For example, tagging techniques (e.g., VLAN) use a tag placed onthe packet to segment forwarding tables to only apply rules associatedwith the tag to a packet. This only segments an existing address space,rather than introducing a new space. As a result, because the addressesare used for entities in both the virtual and physical realms, they haveto be exposed to the physical forwarding tables. As such, the propertyof aggregation that comes from hierarchical address mapping cannot beexploited. In addition, because no new address space is introduced withtagging, all of the virtual contexts must use identical addressingmodels and the virtual address space is limited to being the same as thephysical address space. A further shortcoming of tagging techniques isthe inability to take advantage of mobility through address remapping.

With the packet encapsulated with 2L context information (e.g., theegress port), the 2L logical processing realized by the 1L logicaltables is complete. Next, the process 2100 determines (at 2130) a localfirst level logical egress port in order for the packet to reach thesecond level logical egress port in its encapsulation information. Whenthe packet destination is at a remote domain, this is the logical egressport for the originating first level logical switching element thatconnects to the remote domain containing the destination machine. In theexample of FIG. 19, this is Port 5 of the originating LDPS K 1905. Whenthe packet destination is within the same domain, this will be one ofthe other ports on the originating first level logical switching elementthat connects to the destination machine. The majority of the foregoingdiscussion, however, assumes that the packet destination is in adifferent 1L domain than the packet source.

After determining the 1L logical egress port, the process encapsulates(at 2135) the packet with this local first level logical egress portinformation. That is, the managed switching element prepends informationto the packet (e.g., a logical context) that includes the 1L egress portinformation. As with the 2L encapsulation, the logical contextinformation of some embodiments is a 64-bit tag that includes a 32-bitvirtual routing function field (for representing the logical switchingelement to which the packet belongs (i.e., the originating 1L logicalswitching element)), a 16-bit logical inport field (i.e., the ingressport on the 1L switching element), and a 16-bit logical outport field(i.e., the 1L egress port).

As with the 2L logical egress port, some embodiments only include the 1Llogical egress port within the 1L logical context prepended to thepacket. That is, the logical context that encapsulates the packet doesnot include an explicit tenant ID. Instead, the 1L logical contextcaptures the logical forwarding decision made at the first hop. Fromthis, the first level LDPS ID (i.e., the 1L LDPS from which the packetoriginates) can be determined implicitly at later switching elementswithin the originating domain by examining the logical egress port (asthat logical egress port belongs to the particular 1L logical switchingelement). This results in a flat context identifier, meaning that theswitching element does not have to slice the context ID to determinemultiple pieces of information within the ID. In some embodiments, theegress port is a 32-bit ID.

At this point, the 1L logical forwarding is complete. The process 2100then transmits (at 2140) the twice-encapsulated packet towards thephysical location of the first level logical egress port, and ends. Inthe case of a packet destination outside of the domain, this physicallocation is an interconnection switching element (e.g., an extender or apool node) located at the edge of the domain. In some embodiments, thistransmission actually involves multiple operations. First, the 1Llogical egress port is mapped to a physical address (e.g., the addressof the interconnection switching element, or a port thereupon). Next,this physical address is mapped to a physical port of the managedswitching element so that the packet can be transmitted to the next hop.That is, while the interconnection switching element is the ultimatedestination (within the local 1L domain), there may be one or morephysical switching elements (either managed or unmanaged) in between thesource managed switching element and the interconnecting managedswitching element.

Returning to FIG. 20, the packet 2000 leaves the source hypervisor 2015with a double encapsulation, having the 2L egress context 2020encapsulated inside the 1L egress context 2025. As shown, the packet2000 (destined for a remote machine in a different 1L domain) travelsthrough the local network to a local interconnection switching element2030 (e.g., an extender that connects to the remote 2L domain). Thelocal interconnection switching element 2030 executes the local 1L flow,which removes the local 1L egress context 2025 (as the switching element2030 is the physical realization of the local 1L egress port). The localinterconnection switching element also executes an interconnection flowthat encapsulates the packet 2000 with the ingress context 2035 of theremote 1L logical switching element (i.e., the logical port of theswitching element that connects to the local domain, such as Port 5 oflogical switching element 1910 in FIG. 19).

FIG. 22 conceptually illustrates in greater detail a process 2200 ofsome embodiments for processing packets by an interconnection managedswitching element for a packet exiting the domain of the interconnectionmanaged switching element (e.g., the interconnection switching element2030) in a federated network. This switching element, as stated, may bean extender or a pool node in various embodiments, and containsinformation regarding connections to external networks (including theremote managed network to which the packet is headed). As in the exampleof FIG. 20, in some embodiments only first level processing is performedat the managed switching elements other than the first hop switchingelement.

As shown, the process 2200 begins (at 2205) by receiving a packet at thephysical location of the packet's logical egress port. The logicalegress port of a packet headed out of the network is a particular porton the logical switching element that maps to a particular port on thephysical interconnection managed switching element (e.g., Port 5 of thelogical switching element 1905 of FIG. 19). For an extender, e.g., thisis the physical port that faces the local network (as opposed to one ormore ports facing external networks).

As the packet's outermost encapsulation contains local 1L egress data,the process determines (at 2210) from this encapsulation that the packetis directed to the first level logical egress port (which it has nowreached). As such, the process removes (at 2215) the local first levelencapsulation of the packet. In general, once a packet reaches adestination that maps to a logical egress port in the encapsulation ofthe packet, the switching element at that destination will remove theencapsulation in some embodiments (as it is no longer needed).

With the local 1L processing complete, the process 2200 determines (at2220) the remote first level logical ingress port using interconnectioninstructions contained within the forwarding tables of the managedswitching element. In some embodiments, the 2L network controllercluster passes these interconnection instructions to the local 1Lcontroller, which passes them to the interconnection switching elementwithin the forwarding table information. Specifically, the managedswitching element uses the interconnection instructions to match thelocal logical egress port (from the encapsulation information strippedat 2215) to a remote ingress port. In the context of FIG. 19, Port 5 ofthe originating LDPS 1905 as an egress port matches to Port 5 of thedestination LDPS 1910 as an ingress port.

The process then encapsulates (at 2225) the packet with this remotefirst level logical ingress port information. That is, the managedswitching element prepends information to the packet (outside the 2Legress information, which is unmodified), such as a logical context,that includes the remote 1L ingress port information. This may be a64-bit logical context as described above, in some embodiments, butwithout the 16-bit outport information (which is determined at theremote interconnection switching element). In other embodiments, theremote 1L logical ingress port is prepended to the packet without aspecific LDPS ID, in the same manner as the egress ports describedabove. Just as with the logical egress ports, when the packet reachesthe physical realization of the outermost logical port, theencapsulation will be removed.

At this point, the logical processing for the interconnection switchingelement is complete. The process 2200 then transmits (at 2230) thetwice-encapsulated packet towards the physical location of the remotesite, and ends. Again, this transmission may involve multipleoperations. The managed interconnection switching element maps thelogical ingress port to a physical address (e.g., the address of aremote interconnection switching element), then maps this physicaladdress to a physical port of the managed interconnection switchingelement through which the packet can be transmitted to the next hop. Inthis case, the next hop may be on an unmanaged network (e.g., theInternet) if the two 1L domains are in two different locations (e.g.,data centers in different cities).

In FIG. 20, the packet 2000 is transmitted from the local (originatingnetwork) interconnection switching element through an interconnectingnetwork to a remote interconnection switching element 2040 (e.g., anextender, pool node, etc.). This remote interconnection switchingelement 2040 is located at the edge of the remote 1L domain (e.g., theedge of a remote data center). As shown, when the packet arrives at themanaged switching element 2040, the switching element removes both the1L ingress port information 2035 and the 2L egress port information2020, then adds new 1L egress port information 2045 to the packet 2000.This 1L egress information 2045 indicates the egress port on the local1L logical switching element for the packet (i.e., the port connected tothe destination machine for the packet).

FIG. 23 conceptually illustrates in greater detail a process 2300 ofsome embodiments for processing packets by an interconnecting managedswitching element for a packet entering the domain of theinterconnecting managed switching element (e.g., the interconnectionswitching element 2040) in a federated network. This switching element,as stated, may be an extender or a pool node in various embodiments, andcontains information regarding connections to external networks(including the remote managed network from which the packet isreceived).

As shown, the process 2300 begins by receiving (at 2305) atwice-encapsulated packet at the physical port connected to a remotesite. This packet, as shown in FIG. 20, contains an outer encapsulation2035 identifying the 1L ingress port local to the physical managedswitching element performing the process 2300 as well as an innerencapsulation 2020 identifying the 2L egress port for the packet.

As the packet's outermost encapsulation contains local (to the switchingelement) 1L ingress data, the process determines (at 2310) from thisencapsulation that the packet is directed to the first level logicalingress port (which it has now reached at the interconnecting managedswitching element). As such, the process removes (at 2315) the localfirst level encapsulation from the packet. In some embodiments, whenevera switching element removes encapsulation information from a packet, theswitching element saves that information in a temporary storage (e.g., aregister of the switching element) for the duration of the processing ofthe packet, as the information may be needed for later processing.

With the 1L encapsulation removed, the process determines (at 2320) thesecond level egress port based on the 2L encapsulation of the packet.This is the logical port on the 2L switching element that maps to thepacket destination. The process also removes (at 2325) the second levelencapsulation from the packet. As with the first level encapsulation,some embodiments store this 2L information in a temporary storage (e.g.,a register) for use in any additional processing of the packet.

At this point, the packet has no encapsulation information (at least asrelates to the logical datapath sets of the federated network). Next,the process determines (at 2330) a local first level egress port basedon the second level egress port. The 2L egress port from theencapsulation removed at operation 2325 maps to a particular port in thelocal (destination) 1L logical datapath set. For instance, in FIG. 19,Port 8 of the 2L LDPS 1915 maps to Port 4 of the destination 1L LDPS Z1910. While described here as using the 2L egress information todetermine the 1L egress port, some embodiments also use the destinationaddress contained in the packet headers, as this address also maps tothe 1L egress port.

In addition, some embodiments may execute egress ACL tables within the2L processing at the receiving interconnection switching element. Someembodiments execute such egress ACL tables when the egress port of aparticular logical switching element is removed. Thus, in a single-levellogical network, when a packet reaches the destination managed switchingelement, some embodiments remove the logical context and run the packetthrough egress ACL tables. Similarly, in this case, when the receivinginterconnect removes the 2L processing, it may execute egress ACLtables.

The process 2300 then encapsulates (at 2335) the packet with this localfirst level logical egress port information. As with the encapsulationsperformed at the source managed switching element, in some embodimentsthe encapsulation is in the form of a context tag, such as the 64-bitcontext tag described above. In other embodiments, the context tagcontains only the 1L logical egress port (e.g., the 32-bit egress portdescribed above).

Next, the process transmits (at 2340) the once-encapsulated packettowards the physical location of the local first level egress port. Ingeneral, there will not be any managed switching elements in between theinterconnection managed switching element (usually a non-edge switchingelement, such as an extender or pool node) and the destination managedswitching element (an edge switching element that connects to thedestination machine). As shown in FIG. 11, some embodiments enable adirect connection between the extender and hypervisor for federatednetworks. In other embodiments, an additional managed switching element(e.g., a pool node) may be located between the interconnection switchingelement and the destination managed switching element. In addition, thenetwork may contain intervening unmanaged switching elements. Again,this transmission may involve multiple operations, including mapping the1L logical egress port to a physical destination (the destinationcontained in the packet headers), then mapping this physical address toa physical port of the managed interconnection switching element throughwhich the packet can be transmitted to the next hop.

Returning again to FIG. 20, the packet 2000 is transmitted (with asingle level of encapsulation) through the destination 1L network) to adestination hypervisor 2050. This hypervisor 2050 is the edge managedswitching element to which the destination machine connects(specifically, in this case, the hypervisor on which the destinationmachine operates). As shown, this machine executes its 1L flow torecognize its outport facing the machine as the 1L egress port, removethis egress information 2045, and transmit the packet 2000 (headers andpayload) to the destination machine.

While this example in FIG. 20 (and the subsequent process of FIG. 23)illustrates the removal of the 2L egress port information at the remoteinterconnection switching element 2040 (as it is no longer needed exceptto add the 1L egress information at that switching element), someembodiments do not strip the 2L egress information until the lastmanaged switching element that connects directly to the destinationmachine (i.e., the location of the 2L egress port, such as hypervisor2050). In this case, the interconnection switching element at thedestination domain would remove the 1L ingress information andencapsulate the packet with the 1L egress information, while leaving the2L encapsulation intact.

The previous set of figures described the packet processing performed atseveral of the managed switching elements within a federated network (orset of networks) for a packet originating in one first level network anddestined for a different first level network. FIG. 24 conceptuallyillustrates a different view of the processing performed by a sourcemanaged switching element 2400 (i.e., the managed switching element atwhich a packet arrives after being sent from its source machine, such asthe hypervisor 2015 of FIG. 20). Specifically, FIG. 24 illustratesforwarding table entries 2405 for the source managed switching element2400.

In conjunction with the forwarding table entries, FIG. 24 conceptuallyillustrates the processing pipeline 2450 performed by the source managedswitching element 2400 of some embodiments. As shown by the numbers 1-6,when the managed switching element 2400 receives a packet, the switchingelement uses numerous forwarding table entries to process the packet. Insome embodiments, the physical, 1L, and 2L tables are implemented as asingle table within the managed switching element (e.g., using adispatch port that returns a packet processed by a first entry to theforwarding table for processing by a second entry).

As shown, in this example, a VM 1 is coupled to the managed switchingelement 2400, which is also coupled to a second switching element (whichmay be a managed switching element or an unmanaged switching element).The VM 1 sends a packet 2410 to a destination machine through two levelsof logical switching elements that are implemented by the managedswitching element 2400 as well as other managed switching elements.

The managed switching element 2400 receives the packet 2410 through aninterface of the switching element, and begins processing the packetusing the forwarding tables 2405. The first stage in the processingpipeline 2450 is an ingress context mapping stage 2455 that maps aphysical ingress port (i.e., the interface through which the packet wasreceived from VM1) to a local 1L logical ingress port (i.e., a port ofthe logical switching element implementing the local 1L network thatcorresponds to this interface). As shown by the encircled 1, the managedswitching element identifies a record 1 in the forwarding table thatimplements this ingress context mapping. This record 1 specifies thatthe managed switching element 2400 store the logical context in aregister, or meta field, of the managed switching element. The logicalinport is therefore stored within the switching element for the durationof the packet processing, and can be used in performing additionallookups (e.g., mapping to the next level ingress port). In addition, insome embodiments the logical context (after each of the forwarding tablerecords is applied) indicates the status of the packet within theprocessing pipeline (i.e., in this case, that a first level of ingresscontext mapping has been performed). The record also specifies to sendthe packet to the dispatch port, for additional processing by theforwarding tables 2405.

The second stage in the processing pipeline 2450 is a second ingresscontext mapping operation 2460 that maps the local 1L logical ingressport identified at stage 2455 to a 2L logical ingress port (i.e., a portof the logical switching element implementing the 2L network). As shownby the encircled 2, the source managed switching element 2400 identifiesa record 2 in the forwarding tables 2405 that implements this secondlevel of ingress context mapping. At each of the levels of ingresscontext mapping, as illustrated conceptually by the pipeline 2450, theforwarding entries map a lower level port to a logical port at the nextlevel up. The federated network of some embodiments may have more thantwo levels (e.g., three, four, etc., to allow for traffic engineering atdifferent levels), and in such cases, additional records for performingingress mapping up to each of the levels would be contained within theforwarding tables. The record 2 specifies that the managed switchingelement 2400 store the logical context in a register, or meta field, ofthe managed switching element. The 2L logical inport is therefore storedwithin the switching element for the duration of the packet processing,and can be used in performing additional lookups. The record alsospecifies to send the packet to the dispatch port, for additionalprocessing by the forwarding tables 2405.

Next, the managed switching element 2400 performs the third stage in theprocessing pipeline 2450, the forwarding lookups 2465. These forwardinglookups are illustrated here as a single forwarding table record, butmay in fact involve the use of several records from several tables(e.g., one or more ACL tables, a layer 2 or layer 3 forwarding table,etc.). The hierarchical logical forwarding concept may be applied to anytype of switching/routing network, and therefore any sort of lookups maybe implemented at this level of the processing pipeline. FIG. 25,described below, gives one such example.

The forwarding lookups perform the traditional forwarding decision forthe packet 2410, by identifying an egress port based on the packetdestination. Using this destination contained within the packet header,the forwarding lookups identify a logical port of the 2L switchingelement to which the packet should be sent, as well as any additionalpolicies (e.g., drop, enqueue, etc.) that apply to the packet. Theseforwarding decisions are implemented by the 2L network controller, whichpasses its generated flow records to the 1L network controller (e.g.,via an API exposed by the 1L network controller). The 1L networkcontroller then implements these received instructions within its 1Lflow records, and passes these to the managed switching elements(including the source managed switching element 2400) for application toactual traffic packets. As shown by the encircled 3, a record 3 (whichmay conceptually represent several records) implements these forwardinglookups, and at least specifies that the managed switching element 2400store the 2L egress port in the packet headers (i.e., encapsulate thepacket with the 2L egress context), as well as send the packet to itsdispatch port.

Based on the 2L egress port specified at the third stage of theprocessing pipeline 2450, the managed switching element performs egresscontext mapping 2470 that maps the 2L egress port to an 1L egress port.As this is performed at a managed switching element in the originating1L network, the 2L egress port maps to a port of the local 1L logicalswitching element. For a packet with a destination outside of the local1L network, this will be a logical port used for remote packets (e.g.,Port 5 of the logical switching element 1905 in FIG. 19). As shown bythe encircled 4, the source managed switching element 2400 identifies arecord 4 in the forwarding tables 2405 that implements this egresscontext mapping. The record 4 specifies that the managed switchingelement 2400 store the 1L logical egress port in the packet headers(i.e., encapsulate the packet with the 1L egress context), as well assend the packet to its dispatch port.

Next, the managed switching element 2400 performs another level ofegress context mapping at stage 2475 of the processing pipeline 2450.Whereas the first level of egress context mapping maps a 2L logicalegress port to a 1L logical egress port, this second stage maps the 1Llogical egress port to a physical egress port for the packet within thedomain of the 1L logical network. For a packet traveling to a different1L domain, this is not the physical port of a switching element thatinterfaces directly with the destination machine, but rather thephysical port on an interconnection managed switching element that facesan external network. In some embodiments, the egress context mappingback to the physical level also identifies a port on a next hopintermediate switching element (e.g., by the port's MAC address). Asshown by the encircled 5, the source managed switching element 2400identifies a record 5 in the forwarding tables 2405 that implements thissecond level of egress context mapping. As with the ingress mapping, ina federated network with more than two levels, additional records forperforming egress mapping will be contained within the forwarding tablesfor each of the logical levels. Each level maps an egress port at thehigher level to an egress port at the lower level, down to the physicallevel. In some embodiments, the record 5 specifies that the managedswitching element 2400 store the physical egress port in the packetheaders, as well as send the packet to its dispatch port for furtherprocessing.

Lastly, the managed switching element 2400 performs the physical mappingstage 2480 that specifies a physical port of the managed switchingelement through which to send the (now-modified) packet 2410 in order toreach the physical egress port identified by the egress context mapping(and, therefore, eventually, the destination machine). As shown by theencircled 6, the source managed switching element 2400 identifies arecord 6 in the forwarding tables 2405 that implements this physicalmapping. That is, the record 6 specifies which port of the managedswitching element to send the packet to in order for the packet to reachthe physical address identified at stage 2475 (as opposed to thedispatch port that routes the packet back into the forwarding tables).

The above description for FIG. 24 illustrates the logical pipelineperformed by the first hop managed switching element for a packet in afederated network that is being sent from one 1L logical network to asecond 1L logical network. In some embodiments each managed switchingelement performs the same logical pipeline, though with differentrecords. However, at many of these switching elements, some or all ofthe logical context information is already stored in the packet, andtherefore no actual operation is performed for some of the stages. Forexample, at an intermediate managed switching element between the firsthop and the interconnection switching element, the logical context wouldindicate that the 1L and 2L processing was already complete until the 1Legress port is reached, which does not happen at the intermediateswitching element. At such an intermediate switching element, the onlyoperation actually performed is the physical mapping stage to send thepacket to a port of the physical switching element.

In addition, not all packets within a federated network are necessarilysent from a machine in a first 1L domain to a machine in a different 1Ldomain. Instead, packets will often be sent from a machine in the first1L domain to a different machine in the same 1L domain (e.g., from VM Ato VM B in FIG. 19). In this case, some embodiments nevertheless performthe full processing pipeline 2450, as the actual forwarding decisionsoccur at the 2L level. In such a case, the source managed switchingelement would identify the ingress ports at the 1L and 2L level inexactly the same manner as described above for FIG. 24. The forwardinglookups would still identify an 2L egress port, which would in turn mapto a 1L egress port. The difference, in this case, is that the 1L egressport would not be the port for remotely-destined packets, but rather aport that maps to one of the machines in the 1L domain. As such, thepacket would generally not reach the interconnection switching element,but instead would travel through the local network to the destinationswitching element, which would remove all of the encapsulation and sendthe packet to the destination machine.

As indicated above, FIG. 24 describes a generic processing pipeline.FIG. 25 illustrates a processing pipeline 2550 for a specific type ofnetwork performing OSI layer 2 forwarding (e.g., forwarding based on MACaddress). The pipeline 2550 is the same as the pipeline 2450, exceptthat the forwarding lookups stage 2465 is split into three stages2555-2565. Each of these stages involves a separate forwarding tablelookup, as shown in the forwarding tables 2505 of the managed switchingelement 2500.

The ingress ACL stage 2555 uses the 2L logical ingress port, as well asother fields stored in the packet header (e.g., MAC address, identifierof the 2L logical switching element, etc.) to make a decision about howto proceed with the packet. Some ACL operations include allowing thepacket to continue with further processing, denying the packet (whichwill cause the switching element to discard the packet and cease furtherprocessing), and enqueuing the packet (e.g., sending the packet to aqueue for Quality of Service purposes). The ACL tables may implementother functionalities as well, such as counters (i.e., counting thenumber of packets coming from a particular source address or to adestination address), port security (i.e., only allowing packets comingin through a particular port that originated at a particular machine),and machine isolation (i.e., only sending broadcast/multicast packetsreceived from a particular machine to a particular set of machines).

The layer 2 logical forwarding stage 2560 performs the actual forwardinglookup that determines the logical 2L egress port for the packet. Asthis switching element is performing layer 2 forwarding, the forwardingtable record bases the decision on the destination MAC address of thepacket in some embodiments. Other embodiments performing layer 3 routingmight make a forwarding decision based on the destination IP address ofthe packet.

The third stage of the processing pipeline 2550 performed at the secondlogical level is the egress ACL stage 2565. In general, the egress ACLforwarding table records may include the same operations (e.g., allow,deny, enqueue, etc.) as the ingress ACL forwarding table records, butare based on egress port information or a combination of ingress andegress port information. Some embodiments will only have either aningress ACL table or an egress ACL table, while other embodimentsinclude both tables. In addition, some embodiments may split forwardingor ACL tables into multiple tables. In some embodiments, depending onthe nature of the ACL rules, the number of resulting lookup entries maybe decreased when what could be treated as a single table is split upinto several tables. For instance, certain more complicated ACL rulesmay benefit from such a split of the lookup entries into several tables.

Both FIG. 24 and FIG. 25 illustrate the switching element repeatedlysending packets to a dispatch port, effectively resubmitting the packetback into the switching element. In some embodiments, using softwareswitching elements provides the ability to perform such resubmissions ofpackets. Whereas hardware switching elements generally involve a fixedpipeline (due, in part, to the use of an application-specific integratedcircuit (ASIC) to perform the processing), software switching elementsof some embodiments can extend a packet processing pipeline as long asnecessary, as there is not much of a delay from performing theresubmissions. In addition, some embodiments enable optimization of themultiple lookups for subsequent packets within a single set of relatedpackets (e.g., a single TCP/UDP flow). When the first packet arrives,the managed switching element performs all of the lookups and resubmits(e.g., the 8 forwarding table records illustrated in FIG. 25) in orderto fully process the packet. The switching element then caches the endresult of the decision (in the above case, the addition of theparticular 2L and 1L egress contexts to the packet, and the next-hopforwarding decision out a particular port over a particular tunnel)along with a unique identifier for the packet that will be shared withall other related packets (i.e., a unique identifier for the TCP/UDPflow). Some embodiments push this cached result into the kernel of theswitching element for additional optimization. For additional packetsthat share the unique identifier (i.e., additional packets within thesame flow), the switching element can use the single cached lookup thatspecifies all of the actions to perform on the packet. Once the flow ofpackets is complete (e.g., after a particular amount of time with nosuch packets), in some embodiments the switching element flushes thecache.

D. Additional Aspects of Federated Networks

The above examples illustrate various cases in which two first leveldomains (i.e., first level logical datapath sets) are interconnected viaa second level logical datapath set. However, this is not the onlysituation in which a network operator may use multiple levels ofhierarchical logical datapath sets. For example, there might be morethan two first level domains that are all connected by a second levellogical datapath set, more than two levels of logical datapath sets, oreven one first level logical datapath set split into several secondlevel logical datapath sets.

1. Several 1L Domains

FIG. 26 illustrates a network 2600 with three separate data centers2605, 2610, and 2615. These three data centers 2605-2615 are similar tothe data centers described in FIG. 12, with each center having a set ofend machines (either virtual or physical machines), a set of edgemanaged switching elements, an interconnecting managed switching elementat the edge of the data center, and a first level controller cluster forimplementing first level logical switching elements within the managedswitching elements at the data center.

In this case, the interconnecting managed switching elements form a fullmesh. That is, the switching element 2620 at the first data center 2605connects to the switching element 2625 at the second data center 2610and the switching element 2630 at the third data center 2615 through theexternal network, without having to go through one of the other datacenters. In addition, the managed switching element 2625 connects to theswitching element 2630 through the external network, without travelingthrough the data center 2605.

In addition, the network 2600 includes a second level controller cluster2635. This controller cluster 2635 connects to first level controllerclusters 2640, 2645, and 2650 at the three different data centers. Inthis case, the second level controller 2635 is shown as situated outsideof any of the data centers, but as with the second level controllers inFIGS. 12-14, the controller 2635 may be situated in one of the datacenters, or all three domains could be in a single data center with thesecond level controller in some embodiments.

FIG. 27 illustrates three 1L logical datapath sets (or logical switchingelements) connected by a 2L logical datapath set, along with some of theport mappings performed during packet processing by the logicalswitching elements as implemented in the managed switching elements ofthe network. As shown, this figure illustrates a first 1L LDPS 2705, asecond 1L LDPS 2710, and a third 1L LDPS 2715. Each of these 1L LDPS hasfive ports: three ports for VMs within their local network, and twoports for connecting to the other 1L LDPS. As there are nine machines onthe overall network, the 2L LDPS 2720 includes nine ports.

The various arrows shown in FIG. 27 represent ingress and egress contextmappings performed by forwarding tables of managed switching elementsimplementing the logical switching elements 2705-2720 while processingpackets. For instance, a first arrow 2725 between Port 2 of the 1L LDPS2705 and the 2L LDPS 2720 indicates that switching elements may performboth ingress mappings (from 1L ingress port to 2L ingress port) andegress mappings (from 2L egress port to 1L egress port) between thesetwo ports. On the other hand, the unidirectional arrows 2730 and 2735indicate that switching elements may perform egress context mappingsfrom Port 6 of 2L LDPS 2720 to either Port 4 of 1L LDPS 2705 or Port 5of 1L LDPS 2715, but do not perform ingress mappings from the 1L LDPSports facing external networks to 2L ports. Packets only originate atthese ports from the perspective of the 1L logical switching element(for packets incoming from other domains), but not from the 2Lperspective (from the 2L perspective, the packets always originate at asource machine).

In some embodiments, machines on the 2L network may wish to sendbroadcast packets to all other machines in the network. However, such asetup can create a problem of having broadcast/multicast packetscirculate in an infinite loop. For example, if VM A sends a broadcastpacket, the source hypervisor will identify Port 1 as the logicalingress port (for both the 1L logical switching element 2705 and thenthe 2L logical switching element 2720). As this is a broadcast packet,the egress ports identified by the forwarding tables are Ports 2-9 oflogical switching element 2720. However, three of these egress ports(Ports 4-6) map to Port 4 of the 1L logical switching element 2705 andthree of the ports (Ports 7-9) map to Port 5 of the 1L logical switchingelement 2705, so essentially, the packet is broadcast to each of the 1Lports, two of which connect to remote data centers. When such abroadcast packet reaches the remote 1L logical switching element 2710(via Port 4), this logical switching element will broadcast the packetto all of its other ports, including Port 5. Sending the packet to Port5 of logical switching element 2710 will cause the packet to travel toPort 5 of the logical switching element 2715, which will in turnbroadcast the packet to all of its ports, including Port 4. This sendsthe broadcast packet to Port 5 of the originating logical switchingelement 2705, which broadcasts it to all of its ports. In this way, twoinfinite loops are created, with packets traveling between the networksin both directions. With additional 1L domains, the problem can becomeeven worse.

Accordingly, different embodiments use different solutions to preventsuch an infinite replication issue. For instance, some embodiments flagincoming broadcast packets to prevent such an infinite replicationissue. FIGS. 28 and 29 conceptually illustrate processes of someembodiments for implementing such a flag to prevent broadcastreplication.

FIG. 28 conceptually illustrates a process 2800 of some embodiments forsetting a flag upon receiving a packet. As shown, the process begins byreceiving (at 2805) a packet at a first level logical switching element.In some embodiments, this process is only performed at theinterconnecting managed switching elements (e.g., extenders or poolnodes), because it is only at these switching elements where thereplication problem occurs.

The process determines (at 2810) whether the source of the packet is amachine from a remote first level logical datapath set. This may beaccomplished by checking the header of the packet in some embodiments.Other embodiments instead use the physical interface of theinterconnecting switching element at which the packet was received(i.e., determine whether the packet was received at a port that faces anexternal network.

When the packet source is a remote machine, the process sets (at 2815) aflag in the packet to indicate the remote source. Some embodiments use asingle bit (e.g., a default value of 0, and a value of 1 forremote-source packets). The process then forwards (at 2820) the packettowards one or more destination ports of the first level logicaldatapath set. While this process indicates the determination of whethera packet is a remote source as applied to all packets received at themanaged switching element implementing the process, some embodiments donot apply a test to all packets received at the switching element.Instead, upon receiving a packet on a port that faces an externalnetwork, the switching element automatically applies the remote-sourceflag to the packet.

FIG. 29 conceptually illustrates a process 2900 of some embodiments fordetermining whether to broadcast a packet to all ports of a first levellogical datapath set. As shown, the process begins by receiving (at2905) a packet to broadcast (or multicast) at a first level logicalswitching element. In some embodiments, the process 2900 may beperformed immediately after the process 2800 within a single managedswitching element (i.e., when the process performs the forwardingdecisions at 2820).

The process 2900 determines (at 2910) whether the packet is flagged ashaving a source in a remote first level logical switching element. Asindicated above, this flag may be a single bit that has been set byeither the current or a previous managed switching element in order toprevent infinite replication of broadcast packets in a federatednetwork.

When the packet is flagged as such, the process broadcasts (at 2915) thepacket only to local machines connected to the first level logicalswitching element. That is, in the example of FIG. 27, a broadcastpacket received on Port 4 of the 1L LDPS 2705 with such a flag bit setwould only be sent to Ports 1-3 of the 1L LDPS. On the other hand, whenthe packet is not flagged (e.g., if the packet is received from amachine local to the 1L logical switching element), the processbroadcasts (at 2920) the packet to all ports of the first level logicalswitching element. The process then ends.

This combination of processes will prevent the infinite replicationproblem without over-suppressing packets in the fully connected networksuch as is shown in FIG. 26. However, in some embodiments, the various1L domains may not be fully connected. That is, packets originating in afirst domain with a destination in a second domain may need to travelthrough a third domain. FIG. 30 conceptually illustrates such a network3000. In this case, the first data center 3005 and the third data center3015 do not have a direct connection. Instead, a packet traveling fromthe first data center 3005 to the third data center 3015 would need totravel through the second domain 3010. In this case, the processes shownin FIGS. 28 and 29 will prevent broadcast packets from one of the outerdomains (1L networks 3005 and 3015) from reaching the other one of theouter domain. As such, some embodiments use a time to live (TTL) orother hop counter to prevent the infinite replication problem. That is,each packet can only be replicated either a particular number of timesor for a particular time period before the packet is dropped.

2. Several Hierarchical Levels

While all of the previous examples illustrate two levels of hierarchicallogical switching elements, the principles involved in mapping betweenlogical ingress and egress ports of the different logical levels may beapplied ad infinitum (restricted, of course, by packet sizes, limits ofswitch forwarding tables, and practicality). FIG. 31 illustrates anetwork 3100 with four data centers with three levels of networkcontroller clusters. Specifically, the network 3100 includes a first 1Ldomain 3105, a second 1L domain 3110, a third 1L domain 3115, and afourth 1L domain 3120. The first 1L domain 3105 and the third 1L domain3115 together form a first 2L domain, controlled by a first 2Lcontroller cluster 3125 (which may be located at one of the datacenters, or external to both as shown). The second 1L domain 3110 andthe fourth 1L domain 3120 together form a second 2L domain, controlledby a second 2L controller cluster 3130 (which may be located at one ofthe data centers, or external to both as shown). Finally, the two 2Lcontroller clusters are controlled by a 3L controller cluster 3135, fromwhich flow entries originate in the network 3100. Flow entries originateat the controller cluster 3135 (e.g., in response to user input,detection of a new end machine, etc.), and are pushed down to the 2Lcontroller clusters 3125 and 3130, then from these controllers to the 1Lcontroller clusters and from there to the managed switching elements.

As shown, the first 1L domain 3105 and the third 1L domain 3115 includesthree VMs each, while the second 1L domain 3110 and the fourth 1L domain3120 include two VMs each. FIG. 32 conceptually illustrates the threelevels of logical switching elements implemented for the network 3100,as well as some of the mappings between the ports of these logicalswitching elements.

The first 1L logical switching element 3205 includes three ports for thethree VMs in this domain, as well as a fourth port for packets sent toand received from the other 1L domain within its 2L group (data center3), and a fifth port for packets sent to and received from the other 2Ldomain (irrespective of which 1L domain within the other 2L domain). Theother three 1L logical switching elements 3210, 3215, and 3220 havesimilar port arrangements. Some embodiments, however, include separateports for each 1L domain in other 2L domains. In this case, the 1Llogical switching element 3205 would include six logical ports, as Port5 would be split into two ports, one for each of the 1L domains 3215 and3220.

The first 2L logical switching element 3225 includes six ports for thesix VMs in this domain, as well as a seventh port for packets sent toand received from the other 2L domain. The other 2L logical switchingelement 3230 includes a similar port arrangement, though with only fiveports rather than seven. Finally, the 3L logical switching element 3235includes ten ports, one for each of the VMs in the network. While thisillustrates a three-level hierarchy, in some embodiments the 3L logicalswitching element 3235 could have a port for packets sent to andreceived from another 3L domain, connected together via an 4L logicalswitching element.

This figure also illustrates some of the ingress and egress contextmappings performed by the managed switching elements of the network 3100according to flow instructions received from the network controllers,with ingress mappings flowing from lower levels to higher levels, andegress mappings flowing in the opposite direction. For instance, thearrows 3240 and 3245 indicates that packets sent from VM 2 will bemapped (ingress) from Port 2 of the 1L logical switching element 3205 toPort 2 of the 2L logical switching element 3225, and then to Port 2 ofthe 3L logical switching element 3230, and in the opposite direction foregress mapping of packets directed towards VM 2, once within the domainof the respective switching elements. For packets directed from one ofthe VMs 6, 7, or 8 (within the domain of the 1L logical switchingelement 3210) to VM 2 (or one of VMs 1 and 3), packets will be egressmapped at the source managed switching element to Port 4 of the logicalswitching element 3210. For packets originating at a VM on the 1Llogical switching element 3210 directed towards one of VMs 4, 5, 9, or10, the source managed switching element will map the egress port of the3L logical switching element 3235 (one of Ports 7-10) to Port 7 of the2L logical switching element 3225, and then to Port 5 of the 1L logicalswitching element 3210, as shown by arrows 3250 and 3255.

3. Slicing a First Level Datapath

In the federated network examples illustrated above, multiple firstlevel logical datapath sets are connected together via a single secondlevel logical datapath set. However, in some situations, a single firstlevel logical datapath set might be sliced into several logical datapathsets at the second level. As one example, a service provider might hostnumerous users (e.g., tenants) on its network of virtual machines,exposing a single logical datapath set for each user. However, if a userhas multiple departments, each with their own set of virtual machines,and wants to give these departments separate control of their machines,then some embodiments allow the first level datapath to be sliced intomultiple second level datapaths. In some embodiments, this slicing maybe accomplished using mechanisms present in the more standard federatedcase (e.g., by matching a particular ingress port of an 1L LDPS to aparticular ingress port of a particular 2L LDPS. As will be described,the difference in this case is that different ingress ports on the same1L LDPS will match to different 2L LDPS.

FIG. 33 conceptually illustrates a network 3300 that uses such a slicingapproach. The network 3300 includes three managed switching elements forconnecting eight virtual machines within a data center 3305. The datacenter 3305 includes a single first level network controller cluster3310. Connected to this first level controller cluster 3310 are twoseparate second level controller clusters 3315 and 3320. These secondlevel clusters each separately generate flow entries that are pusheddown to the single first level controller 3310. The first levelcontroller includes these received flows within the first level flowspushed to the managed switching elements. Within these first level flowsare lookups that identify which ports of the first level logicalswitching element match to the different second level logical switchingelements.

FIG. 34 conceptually illustrates example logical switching elements forthe network 3300. In this case, only four of the eight VMs shown arepart of the logical network. Thus, the 1L LDPS 3405 has four ports, onefor each of these VMs. Because there are no VMs in the network externalto this 1L domain, the 1L logical switching element 3405 has only thesefour ports, and no ports facing an external logical network. Each of thesecond level logical switching elements has two ports—VMs 1 and 5 are onthe first 2L logical switching element 3410, while VMs 2 and 8 are onthe second 2L logical switching element 3415. While VMs 1 and 8 are ontwo different second level logical switching elements, they are on thesame first level logical switching element and therefore couldtheoretically exchange packets within the 1L domain. However, if theowner (e.g., department) of the 2L logical datapath set 3410 wished toisolate its VMs from those on other 2L datapaths, this could be easilyimplemented through the 2L ACL tables (e.g., using port security andmachine isolation techniques), which are sent to the 1L controllercluster and implemented by the managed switching elements.

In some embodiments, enabling the 1L logical datapath set to be used asa platform for multiple 2L logical datapath sets requires similartechniques to the use of multiple 1L LDPS on a physical network. Thatis, just as the 1L flows require ingress context matching to identifythat a particular physical machine belongs to a particular 1L LDPS, the2L flows require ingress context matching to identify that a particularport of an 1L LDPS belongs to a particular 2L LDPS, as opposed to other2L LDPS. This requires that the 1L LDPS has to support such matching inits logical pipeline abstraction.

For a typical single-level (i.e., non-federated) logical network, insome embodiments the packet processing in the 1L LDPS's operationsrequires matching over packet headers, as well as the slice. Therefore,the physical switching element needs to support matching over a 1L LDPSslice/context ID, and packet headers. In general, implementing a secondlevel logical datapath set on top of this requires the physicalswitching element to support matching over the 1L LDPS slice/context ID,a 2L slice/context ID, and the packet headers. This can be implementedas a nested structure, with each 1L LDPS slice holding a set of 2L LDPSslices.

In the runtime packet processing, initially there is no slicing. When amanaged switching element receives a packet, the switching elementinitially identifies the first slice (1L LDPS). In some embodiments, theswitching element loads the 1L LDPS context/slice id into a register andscopes subsequent matches into the 1L LDPS. While doing this, theswitching element removes the frontmost encapsulation header and savesany information into a register for the 1L LDPS. The 1L LDPS thenproceeds to its logical lookup tables (though there may be additionallookups at this level, not discussed above). These lookups identify the2L LDPS and save this result into another register in some embodiments.In addition, the managed switching element removes the frontmostencapsulation header and saves any information into a register for the2L LDPS. Next, the matches are scoped into the 1L and 2L LDPS. After thevarious 2L LDPS lookups (e.g., ACL tables, forwarding, etc.) arecomplete, the packet begins traversing back towards the physical: once2L LDPS processing finishes, the managed switching element saves itsforwarding decision into a packet header by adding a new encapsulation.Similarly, once the 1L LDPS processing finishes, the switching elementsaves this result into the header by adding an encapsulation to thefront of packet.

III. Interconnecting Disparate Networks

The above section describes the interconnection of managed networks(e.g., networks such as those described in Section I). In some cases,however, a network manager will want to provide connections between afirst network segmented using a first tagging or tunneling technique(e.g., VLAN, Mac-in-Mac, L2 over L3, MPLS, etc.) and a second networksegmented using a second tagging or tunneling technique. Even when thetwo networks use the same technique (e.g., both networks use VLANs), theimplementation of that technique (e.g., the structure of the tags usedin packet headers) may be different between the two networks such thatthey are effectively using two different techniques.

Some embodiments provide a mechanism for connecting such disparatenetworks across a common interconnecting network (e.g., an L3 network)that can forward traffic between the disparate networks. To connect suchnetworks, some embodiments use a single managed interconnectionswitching element (e.g., an extender) at the edge of each of thesegmented networks, then manage these interconnection switching elementswith a network controller cluster that defines a logical datapath setbetween the sites.

A. Network Structure

1. Single Logical Layer

FIG. 35 conceptually illustrates three separate segmented networks 3505,3510, and 3515. The networks 3505 and 3515 are each segmented into twoVLANs using VLAN tagging (network 3505 segmented into VLANs 3520 and3525, and network 3515 segmented into VLANs 3540 and 3545), while thenetwork 3510 is segmented into two labels 3530 and 3535 using MPLSlabeling. The figure illustrates that these networks 3505-3515 are eachconnected to an L3 network 3550. However, simply being connected to acommon physical network does not necessarily provide the end machines inthe different segments with the ability to communicate with each other.

These three separate networks could be all owned and operated completelyindependently, without requiring any interconnection. However, theowner(s) of either the physical networks or the virtual machines on thenetworks might wish for their machines to have the ability tocommunicate with each other. For instance, if a first organization thatowns the segmented network 3505 and uses a first type of VLAN taggingmerges with a second organization that owns the segmented network 3515,the merged organization might want their IT departments (e.g., VLAN 13520 and VLAN 4 3545) to be able to communicate as though all themachines were on a single L2 network.

In order to enable communication between network segments at differentlocations, some embodiments locate an interconnection switching element,such as an extender, at the edge of each of the segmented networks, thenmanage these interconnection switching elements with a networkcontroller cluster. FIG. 36 conceptually illustrates such a solution forthe networks 3505-3515. This figure illustrates that threeinterconnection managed switching elements 3605, 3610, and 3615 havebeen located at the edges of the three segmented networks 3505-3515.

The three interconnecting managed switching elements 3605-3615 aremanaged by a network controller cluster 3620, which may be a singlenetwork controller or several controllers that share information. Thiscontroller cluster is similar to those described above (e.g., with acontrol application and virtualization application). However, where theports of the logical switching element defined by a network controllerinstance 610 or 1215 face end machines (either virtual or physical), theports of the logical switching element defined by the controller 3620face network segments (e.g., a particular VLAN at a particular site).

FIG. 37 conceptually illustrates a logical switching element 3700defined by the network controller cluster 3620 and implemented by thethree interconnecting managed switching elements 3605-3615. As shown,each of the ports of the logical switching element 3700 faces one of thenetwork segments from FIG. 35. The VLAN 1 3520 connects to Port 1, theMPLS Label 2 3535 connects to Port 2, and the VLAN 3 3540 connects toPort 3 of the logical switching element 3700. The interconnectionswitching elements 3605-3615 that implement the logical switchingelement 3700 store forwarding tables that include flow entries pusheddown from the network controller that specify how to remove networksegmentation headers (e.g., VLAN tags, MPLS labels, etc.) for outgoingpackets, apply security policies, forward packets between theinterconnection switching elements, and insert network segmentationheaders for incoming packets. The details of how these flow entries aregenerated by the network controller and pushed down to managed switchingelements, as well as the details of processing packets by the managedswitching elements, are described in the sections below.

FIG. 38 conceptually illustrates information stored in a managedswitching element for interconnecting segmented networks. Specifically,this figure shows information stored in the tables of the managedswitching element 3605, at the edge of the segmented network 3505. Themanaged switching element tables include decoding information 3805,logical forwarding information 3810, and egress mapping information3815.

The decoding information 3805, along with additional information (e.g.,the port on which a packet arrived), enables the managed interconnectionswitching element to determine the logical context of a packet exitingthe local segmented network (i.e., the segmented network at the edge ofwhich the switching element is located). In this situation, the decodinginformation 3805 stores information regarding the VLAN 1 context taginformation, which allows the interconnection switching element toidentify a packet as originating at a machine in VLAN 1. In someembodiments, this decoding information also specifies how to add VLAN 1information for packets incoming to network 3605 from a remote network,and destined for a machine on VLAN 1.

The logical forwarding information 3810 implements the logical switchingelement 3700, enabling the switching element 3605 to perform logicalforwarding decisions between the different networks on the logicalswitching element. As shown, the information 3810 includes lists ofdestination addresses for the different segmented networks connected tothe logical switching element 3700. The switching element uses thisinformation to identify a logical port to which a packet should beforwarded (by identifying the destination address in one of the lists).In addition, any ACL tables for implementing security policy arecontained within the logical forwarding tables 3810.

Finally, the switching element stores egress mapping and delivery tables3815, which specifies how a packet with a given logical destination(e.g., VLAN 3) is delivered to a next hop. This information includestunnel information (e.g., specifying tunnels defined across theinterconnecting network between the different managed switchingelements) as well as physical port mapping information to identify thephysical port of the switching element through which an outgoing packetshould be sent.

Just as the physical switching elements in a data center may implementseveral logical switching elements simultaneously (e.g., betweendifferent users' VMs), the physical managed switching elementsinterconnecting different segments of networks may also implementseveral logical switching elements. FIG. 39 illustrates a scenario forthe networks 3505-3515 in which two different network controllers 3620and 3920 generate flow entries for two different logical networks, andpush the flows to the same switching elements 3605-3615. As shown, inthis case both of the controllers connect to all three switchingelements, but in some embodiments a first network controller clustermight control first, second, and third switching elements at the edgesof first, second, and third networks, while a second network controllercluster controls the first switching element as well as a fourthswitching element at the edge of a fourth network, or other similarpossibilities.

FIG. 40 conceptually illustrates the logical switching elements 3700 and4000 defined by the network controller clusters 3620 and 3920,respectively, and implemented by the three interconnecting managedswitching elements 3605-3615. The logical switching element 3700 isdescribed above. The logical switching element 4000 also has three portsthat each face a network segment from FIG. 35. In this case, the VLAN 23525 connects to Port 1, the MPLS Label 1 3530 connects to Port 2, andthe VLAN 4 3545 connects to Port 3 of the logical switching element4000.

FIG. 41 conceptually illustrates information stored in the managedswitching element 3605 for interconnecting the segmented networks on thetwo logical switching elements 3700 and 4000. In this case, theswitching element essentially stores two sets of information. Thedecoding information 4105 includes context tag information for both ofthe VLANs in the network 3505, as both are on logical datapath sets.Similarly, the logical forwarding information 4110 implements both ofthe logical switching elements 3700 and 4000, including all of theinformation shown in the forwarding tables 3810 as well as correspondinginformation implementing the logical switching element 4000. The flowentries for these tables are received separately from the two differentnetwork controllers, and are also generated separately (from thereceived physical control plane data to the stored physical forwardingplane data) by the managed switching element.

Finally, the egress mapping and delivery tables 4115 includes someoverlapping information used for transmitting packets on either of thetwo logical switching elements in some embodiments. For instance,packets may use the same tunnels between the physical switching elementsirrespective of to which logical switching element the packet isassigned.

2. Several Logical Layers

The above examples illustrate a situation in which several segmentednetworks are connected at a single logical layer. That is, a singlelogical switching element is defined by a network controller cluster,and this logical switching element contains a single port for each ofthe network segments. In some embodiments, the principles describedabove in Section II, for hierarchically arranging networks usingmultiple levels of controller clusters and logical datapath sets, can beapplied to the problem of interconnecting several segmented networks.

FIG. 42 conceptually illustrates four segmented networks 4205-4220 (forsimplicity, illustrating only one VLAN at each network), connected usingmultiple logical layers. As shown, the network 4205 contains a VLAN 14225, and has a managed interconnecting switching element 4245 at itsedge. The network 4210 contains a VLAN 2 4230, and has a managedinterconnecting switching element 4250 at its edge. The network 4215contains a VLAN 3 4235, and has a managed interconnecting switchingelement 4255 at its edge. Finally, the network 4220 contains a VLAN 44240, and has a managed interconnecting switching element 4260 at itsedge.

In this example, however, there is no single controller cluster thatconnects to each of the physical switching elements 4245-4260 togenerate the flows for those switching elements. Instead, the first andsecond switching elements 4245 and 4250 are controlled by a firstfirst-level controller cluster 4265 and the third and fourth switchingelements 4255 and 4260 are controlled by a second first-level controllercluster 4270. As such, a first logical switching element routes packetsbetween machines on VLAN 1 4225 and machines on VLAN 2 4230, while asecond logical switching element routes packets between machines on VLAN3 4235 and machines on VLAN 4 4240.

In addition, the provider of the interconnecting services (e.g., theowner of the four networks being interconnected) may want the ability toconnect all four of the VLANs 4225-4240 together. As such, a secondlevel controller cluster 4275 is introduced in order to handle connectthe two logical datapath sets defined by the first level controllers. Asin the case of federation between managed networks described above, thesecond level controller cluster 4275 generates flow entries defining thesecond level logical switching element, that are passed down to the twofirst level controller clusters 4265 and 4270, which generate their ownflow entries (that implement the flows received from the second levelcontroller cluster) and pass these flows to their respective managedinterconnection switching elements.

FIG. 43 conceptually illustrates the three logical datapath sets definedby the three network controller clusters of FIG. 42. Specifically, thefigure illustrates two 1L logical datapath sets 4305 and 4310, with aninterconnecting 2L logical datapath set 4315. The first level logicaldatapath set 4305 defined by the first 1L controller cluster 4265includes three ports: one port for each of the two VLANs beingconnected, and a third port for the remote network segments (VLANs 4235and 4240). Similarly, the first level logical datapath set 4310 definedby the second 1L controller cluster 4270 includes three ports as well:one for each of the VLANs connected on its half of the overall network,and a third port for the remote network segments (VLANs 1 and 2). The 2Llogical datapath set 4315, meanwhile, includes only four ports, one foreach of the network segments.

This second interconnection model, applying the federated networkconcept to the interconnecting switching elements, enables more robusttraffic engineering. While not necessarily as important in theillustrated example (with only four total networks being connected, thehierarchical approach enables more intelligent traffic engineering,especially in larger-scale scenarios. In the single-level model, a fullmesh of tunnels between the managed switching elements are defined, andthis requires a reliable connectivity for each such connection.Especially as the number of networks being connected increases, and thenetworks move further apart physically, this may result inoverprovisioning of the network capacity. Furthermore, as the packetsare encapsulated in these tunnels, traffic engineering to enforcepolicies is difficult. To perform such engineering, the intermediateswitching elements would need to look inside the encapsulation in orderto make decisions.

In the hierarchical approach, the second level network controllercluster receives traffic engineering policies from user settings, anduses optimization routines combined with network statistics receivedfrom the lower-level controllers. In some embodiments, the lower-levelcontrollers automatically collect information regarding their respectiveregions of the interconnecting network (i.e., from the switchingelements implementing their lower-level logical datapath set), and passthis information upward to the second level network controller. In someembodiments, these statistics relate to the number of packets being sentout and received at various ports of the managed switching elements, thetravel time of the packets along different paths, etc.

Irrespective of whether any specific policies have been set by users,the second level controller uses the statistics as an input into anoptimization routine that determines in which direction traffic shouldbe routed. For example, a packet sent from a machine at a networksegment on a first 1L logical datapath set to a machine at a networksegment on a second 1L logical datapath set might have several differentpaths it could take to arrive at the second 1L logical datapath set.Based on the network statistics, the second level controller determinesan optimal one of these several different paths (which might travelthrough different sets of intervening 1L logical datapath sets) forpackets to travel along from the first to the second 1L logical datapathset.

The optimization routine may also be affected by user settings in someembodiments. The user settings might specify that a certain quality ofservice (QoS) is required for packets sent from a particular machine,from a particular network segment, to a particular destination machineor network segment, etc. User settings can also specify that eitherpackets from/to a particular source/destination should always be sentalong a particular path (overriding the optimization processing) orshould always receive the best connection. The user can specifydifferent classes of traffic as well. For packets either given a lowerclass, or not addressed (i.e., with no guaranteed QoS), the optimizationroutines of some embodiments attempt to send the packets over the bestconnection without interrupting higher class traffic.

B. Generation of Flow Entries

In a single level interconnected network of some embodiments, flowentries are generated by the network controller and pushed down to themanaged switching elements, as would be the case in a typical managednetwork within a data center. That is, a network controller of someembodiments includes (i) a control application that receives logicalcontrol plane data and converts this to logical forwarding plane dataand (ii) a virtualization application that receives logical forwardingplane data and converts this to physical control plane data, which ispushed to the managed interconnection switching elements. In someembodiments, each of these sets of data (logical control plane, logicalforwarding plane, physical control plane) are stored in the networkcontroller as nLog tables, and the control and virtualizationapplications perform nLog table mapping operations to convert from onedata plane to the next using an nLog rules engine. In some embodiments,in fact, the control application and virtualization application use thesame rules engine to perform their table mappings. As mentionedpreviously, the network controller of some embodiments is described ingreater detail in U.S. application Ser. No. 13/177,533, incorporated byreference above.

However, some differences do emerge to account for the fact that thelogical ports correspond to network segments (which may have numerousassociated addresses) rather than single machines. When a new VM isadded or moved within a network, some embodiments require new logicalflow entries, while other embodiments do not. This situation isdescribed in subsection D below, regarding learning in theinterconnected network of some embodiments.

When a user specifies that a new network segment be added to theinterconnected network, the control application receives an event that aparticular segment is at a new logical port (“Port Y”). This event isanalogous to the addition of a new machine in a typical managed network,such as is described above in Section I. The control application thentranslates this event into an update to the logical lookup table thatinstructs a switching element to forward a packet to the new logicalPort Y when the packet destination matches with the new network segment(i.e., when the destination address belongs to the particular networksegment). The virtualization application then generates a physicalforwarding plane lookup that adds another layer of matching. This newlookup states that if the 2L logical datapath set is matched (i.e., thesource network segment is on the logical switching element) and that thedestination address matches the new network segment, then the packetshould be forwarded to the new logical Port Y. The virtualizationapplication then pushes this lookup entry to the differentinterconnection switching elements.

FIG. 44 conceptually illustrates a process 4400 of some embodimentsperformed by the network controller for an interconnecting network inorder to generate new flow entries for an event detected at the logicalcontrol plane. As shown, the process 4400 begins by receiving (at 4405)an update to the logical control plane. Such an update may be a userentering a particular ACL policy (e.g., enabling port security ormachine isolation at a particular port, requiring a particular QoS for aparticular machine, etc.). In addition, updates pushed upwards from themanaged switching elements may be received at the control plane. Whenthe user configures a new network segment (e.g., a new VLAN) to attachto the network, this generates an event. In addition, when a new machine(e.g., a new MAC address) is detected within one of the network segmentsby one of the interconnecting switching elements, some embodiments pushthis information up to the network controller.

The process then determines (at 4410) whether the update to the logicalcontrol plane requires the creation of new flow entries. For instance,if the update simply indicates that the network state has not changed,then no new entries are required, and the process ends. On the otherhand, if the update specifies new ACL rules, or indicates the attachmentof a new network segment to the network, then the network controllerwill begin generating new flow entries. In addition, as described below,different embodiments use different learning models to distribute theMAC addresses associated with each port (as each port will have severalpossible destination MAC addresses).

Next, when the update requires new entries, the process 4400 translates(at 4415) the logical control plane update into an update to the lookupsin the logical forwarding plane. In some embodiments, this translationinvolves turning a piece of data into a lookup entry. For instance, if anew network segment (VLAN Q) is attached to a logical Port Z of thelogical switching element for the interconnecting network, then thelogical control plane states “VLAN Q is at Port Z”. Using tables thatlist the MAC addresses associated with VLAN Q, the control applicationtranslates this update into an update to the logical forwarding planelookup table that reads “If destination matches MAC {A}→Forward to PortZ”. In this case, {A} is the set of MAC addresses associated with VLANQ. Some embodiments create this forwarding plane lookup as a separateentries for each MAC address in the set {A}

In addition, as with the multi-level network described above, thegeneration of ACL table entries will also be described. As with theentries in Section II, the conversions tend to be feature agnostic(again, primarily involving the addition of match conditions). Thus, atthe logical control plane, the user specifies that the Port Z should besecured—that is, that network data entering and exiting the logicalswitching element through the particular port have only certainaddresses that the switching element has restricted the port to use.However, these entries differ somewhat from the example given for thefederated network above, because each port will have numerous MACaddresses/IP addresses with which it is associated. For example, if aparticular VLAN has 1000 machines running on it, then the port securityentry for that VLAN will need to include 1000 different MAC addressesand IP addresses as allowed. For instance, the logical forwarding planeentry for the ingress ACL will state “If received from Ingress PortZ→Allow, or Drop If ARP MAC not {A} or IP not {B}, or Drop If MAC not{A} or IP not {B}. In this case, {A} is the set of MAC addressesassociated with VLAN Q (as stated above), and {B} is the set of IPaddresses at VLAN Q. The egress ACL lookup prevents packets not sent toa correct address from exiting the switching element at Port Z onto theVLAN Q, with an instruction of “If sent to Egress Port Z→Drop If dest.IP not {B}”. This prevents other IP addresses from being used within theVLAN Q; by the nature of the logical forwarding, packets sent to MACaddresses not in the set {A} will not be directed to Port Z in the firstplace. In addition, some embodiments also mandate a match of the MACaddress to the IP address in the egress ACL entry. That is, the entryrequires that “If sent to Egress Port Z and match MAC A_(N)→Drop Ifdest. IP not B_(N)”, where A_(N) and B_(N) are matching MAC and IPaddresses. This ensures that a first VM within the VLAN Q does not usethe IP address of a different VM within the same VLAN.

After translating the logical control plane update into logicalforwarding plane lookups, the process then translates (at 4420) thelogical forwarding plane data into physical control plane lookups. Aswith the logical control plane to logical forwarding plane translation,in some embodiments the conversion from logical forwarding plane tophysical control plane is performed as an nLog table mapping operation(e.g., using the same table mapping engine as for the logical controlplane to logical forwarding plane operation). For both the ACL lookupsand the attachment of a new machine, the virtualization application addsa match of the logical datapath set to the entry. These conversions arethe same as for a single level network within a datacenter. Thus, thefirst entry (to attach a new VLAN Q) now states “If match LDPS and Ifdestination matches MAC {A}→Forward to Port Z”. Similarly, the ingressACL entry reads “If match LDPS and If received from Ingress PortZ→Allow, or Drop If ARP MAC not {A} or IP not {B}, or Drop If MAC not{A} or IP not {B}”, and the egress ACL reads “If match LDPS and If sentto Egress Port Z→Drop If dest. IP not {B}”.

In addition to translating the logical forwarding lookups to physicalcontrol plane lookups, the process 4400 also generates (at 4425)additional physical control plane entries in order to realize thelogical forwarding plane over the physical network. In some embodiments,the virtualization application rules engine creates additional flowentries to handle the operations around the forwarding lookups. Asdescribed in the federating case, these lookups include ingress portintegration, egress port integration, and tunnel sending and receivingentries.

In some embodiments, these lookups are generated as soon as the newnetwork segment (e.g., VLAN Q) is added to the interconnecting networkat a particular physical port. When ACL entries are subsequentlygenerated for the particular port, these additional physical controlplane entries are not affected. For the sake of the ongoing example, thephysical port to which the network containing VLAN Q connects is Port Wof the extender located at the edge of the segmented network. For thisexample, the ingress port integration entry matches the physical Port W(and the VLAN Q) to the logical ingress port Z. Thus, this entry states“If received from physical ingress Port W and match VLAN Q→Mark logicalingress as Port Z”. The additional match over VLAN Q enables theinterconnection switching element to differentiate between severalnetwork segments located on the same network, based on the segmentcontext headers identified in the packet.

The egress port integration entry matches the forwarding decision at thelogical level to a physical port. Specifically, the virtualizationapplication generates an entry that states “If sent to Port Z→Runthrough egress pipeline then send to Port W”. As for the flows describedin Section II, these entries are sent to the managed interconnectionswitching element to which VLAN Q connects. However, for a packet to besent out onto VLAN Q, the interconnection switching element will alsoneed to append the VLAN context tag. Accordingly, some embodimentsmodify the egress port integration lookup to send to the particularinterconnection switching element, so that the lookup states “If sent toPort Z→Run through egress pipeline then append VLAN Q context then sendto Port W”. In other embodiments, the appending of the network segmentcontext is specified in a separate lookup entry.

In addition, as with the federated network described above, someembodiments also generate lookups to handle the receipt and transmissionof packets across the different interconnection switching elements. Inthe full mesh case, there is a separate tunnel between each pair ofinterconnection switching elements, and thus the only interconnectionswitching element that receives a packet for a tunnel is theinterconnection switching element at the destination network for thepacket. Thus, the other interconnection switching elements get aphysical control plane lookup of “If sent to Port Z→Encapsulate with Z'scontext ID and output to physical port via tunnel that connects todestination switch”. For the receiving side of the tunnel, at themanaged switching element that contains Port W, the virtualizationapplication generates a tunnel decapsulation lookup entry that states“If tunneled→Decapsulate to identify logical port, then Resubmit”. Theresubmission results in the execution of the egress port integrationdescribed above.

With all of the physical control plane entries generated, the process4400 identifies (at 4430) the managed switching elements to receive thegenerated lookups. As described for some of the various lookup entriesgenerated at 4420 and 4425, not all of the managed interconnectionswitching elements will receive every lookup. For example, the tunnelsending lookups will not be sent to the managed switching element towhich the network segment actually connects, while the tunnel receivingand port integration lookups are only sent to that managed switchingelement. Furthermore, except in the rare situation that two of thenetwork segments on the same logical switching element are also in thesame segmented network (e.g., VLAN 1 and VLAN 2 of a particularVLAN-segmented network), the actual forwarding lookups will not need tobe sent to the interconnection switching element to which thedestination network segment is connected.

Finally, the process 4400 pushes (at 4435) the generated flow entries tothe identified interconnection switching elements, then ends. In someembodiments, the network controller communicates directly with themanaged interconnection switching elements. However, in otherembodiments, the network controller that performs the conversion of thelogical forwarding plane data into the physical control plane data sendsphysical control plane data to master controllers for the particularinterconnection switching elements that are to receive the data, andthese master controllers push the data to the switching elements. Inaddition, while this example describes the computation of physicalcontrol plane data customized for particular switching elements (e.g.,with port numbers of the particular switching elements), someembodiments compute universal physical control plane data that isgeneric to any particular switching element. In this case, either themaster controller or a chassis controller at the managed interconnectionswitching element performs the conversion to customized physical controlplane data for the managed switching elements. In some embodiments, the1L controller propagates the generated flow entries (e.g., to the mastercontroller, from the master controller to the managed switchingelements) through an object-oriented (NIB) data structure, while otherembodiments use direct communication channels (e.g., RPC calls, OpenFlowentries, updates over the configuration protocol) to exchange the flowentries.

FIG. 45 conceptually illustrates some of these input and output tablesthrough the various flow generation operations of some embodiments.Specifically, FIG. 45 conceptually illustrates the input and outputtables for a control application 4505 and a virtualization application4510 of a network controller that manages an interconnecting network. Asshown, the control application 4505 includes an API 4515, input tables4520, a rules engine 4525, output tables 4530, and a publisher 4535.

The API 4515 provides an interface for translating input into thecontrol plane input tables 4520. This API 4515 may be used by varioustypes of management tools with which the user can view/and or modify thestate of a logical network (in this case, network for interconnectingnetwork segments). In some embodiments, the management tools provide auser interface such as a graphical user interface that allows a visualconfiguration of port bindings, ACL rules, etc. (e.g., through a webbrowser). Alternatively, or in conjunction with the graphical userinterface, some embodiments provide the user with a command line tool orother type of user interface.

Based on the information received through the API, as well as updates tothe network state received from the managed switching elements (notshown), the control application generates the input tables 4520. Theinput tables represent the state of the logical switching elementsmanaged by the user in some embodiments. As shown in this figure, someof the input tables include the association of sets of MAC addresses/IPaddresses that are part of a particular network segment with logicalports of the logical switching element, as well as ACL rules set by theuser. In this case, the Port Z is associated with VLAN Q, which includesthe MAC addresses {A} and IP addresses {B} and is secured. In someembodiments, the input tables also include information on the contexttags (e.g., for VLAN Q).

The rules engine 4525 of some embodiments performs various combinationsof database operations on different sets of input tables 4520 topopulate and/or modify different sets of output tables 4530. Asdescribed in further detail in U.S. application Ser. No. 13/288,908,incorporated by reference above, in some embodiments the rules engine isan nLog table mapping engine that maps a first set of nLog tables into asecond set of nLog tables. The output tables 4540 populated by the rulesengine of the control application 4505 include logical forwarding planelookups (e.g., mapping a set of MAC address to a destination outputport) and logical forwarding plane ACL entries (e.g., securing Port X).

The publisher 4535 is also described in further detail in U.S.application Ser. No. 13/288,908, and publishes or sends the outputtables 4530 to the virtualization application 4510, in order for thevirtualization application to use the output tables 4530 among its inputtables. In some embodiments, the publisher 4535 also outputs the tablesto an object-oriented data structure (NIB) that stores network stateinformation.

The virtualization application 4510 receives the output tables 4530 ofthe control application 4505, and converts this logical forwarding planedata to physical control plane data to be sent to the managedinterconnection switching elements. As shown, the 2L virtualizationapplication 4510 includes a subscriber 4540, input tables 4545, a rulesengine 4550, output tables 4555, and a publisher 4560. The subscriber4540 of some embodiments is responsible for retrieving tables publishedby the publisher 4535 of the control application 4505. In someembodiments, the subscriber 4540 retrieves these tables from the sameobject-oriented data structure to which the publisher stores the tableinformation. In other embodiments, a change in the tables is detected bythe virtualization application in order to initiate the processing.

The input tables 4545 include, in some embodiments, at least some of theoutput tables 4530, in addition to other tables. As shown, in additionto the logical forwarding plane data generated by the controlapplication 4505, the input tables 4545 include additional port bindinginformation (matching logical ports with physical ports of particularmanaged interconnection switching elements). In addition, someembodiments include tunnel information that describes the tunnelsbetween the various interconnection switching elements (e.g.,extenders). In some embodiments, additional pathway information is notneeded, because the interconnection switching elements form a full mesh.

In some embodiments, the rules engine 4550 is the same as the rulesengine 4525. That is, the control application 4505 and thevirtualization application 4510 actually use the same rules engine insome embodiments. As indicated, the rules engine performs variouscombinations of database operations on different sets of input tables4545 to populate and/or modify different sets of output tables 4555. Insome embodiments, the rules engine is an nLog table mapping engine thatmaps a first set of nLog tables into a second set of nLog tables. Theoutput tables 4555 populated by the rules engine 4550 include physicalcontrol plane lookups (e.g., mapping a set of MAC Addresses to adestination logical port when the LDPS is matched) and physical controlplane ACL entries (e.g., securing Port X). In addition, the ingress andegress port integration and tunnel sending/receiving lookups aregenerated by the rules engine 4550 in some embodiments. In addition tothe information shown in the figure, some embodiments also include inthe output tables the correct managed switching elements to receive thedifferent tables.

Finally, the publisher 4560 is similar to the publisher 4535 in someembodiments. The publisher 4560 publishes and/or sends the output tables4555 to the managed interconnection switching elements that implementthe logical network between network segments. In some embodiments, thesemanaged interconnection switching elements are all extenders, though inother embodiments other types of managed switching elements may beincluded (e.g., pool nodes). In some embodiments, the publisher 4560outputs the tables to an object-oriented data structure (NIB) thatstores network state information.

One of ordinary skill in the art will recognize that the input andoutput tables shown in this figure are simplified conceptualrepresentations of the actual tables, which are generated in a databaselanguage appropriate for the rules engine (e.g., nLog) and may provideadditional information to that shown. Furthermore, different embodimentswill use different sets of tables. For instance, the logical portaddress and port binding tables of some embodiments are actually asingle table that binds a particular set of MAC and IP addresses in anetwork segment behind a particular physical port of a particularextender to a particular logical port.

The foregoing discussion in this section related to the flat full meshinterconnection network. As indicated, however, some embodiments applythe principles of federated networks to the interconnecting network. Insuch a scenario, the flow generation processes take place at multiplelevels of controllers, as described in Section II above. In addition,the modifications related to having numerous destination MAC addressesat each of the logical ports, and the need to remove/add network segmentcontext tags will be accounted for. Thus, the control plane (at the 2Lnetwork controller) will state that a particular network segment bindsto a particular 2L port and a particular 1L port. The forwarding lookupsare generated as described above, from the 2L controller to the 1Lcontroller, resulting in a physical control plane lookup of “If match 1LLDPS and if match 2L LDPS and if destination matches NetworkSegment→Forward to 2L Port”. In addition, the 2L controller generatesentries for 1L←→2L port mapping, and for the tunnels between the 1Ldomains. Similarly, each 1L controller generates entries forphysical←→1L port mapping, and the tunnels within its 1L domain.

In some embodiments, the pathways between 1L domains are determined bythe 2L network controller. For instance, a machine in a first networksegment located in a first domain might send a packet to a machine in asecond network segment located in a second domain that does not have adirect connection to the first domain (i.e., there is no tunnel definedbetween any extender in the first domain and any extender in the seconddomain). In addition, in some situations, multiple possible pathsthrough the 1L domains are possible in order for the packet to reach itsdestination. For example, the packet might be able to travel eitherthrough a third domain or a combination of a fourth domain and a fifthdomain.

The determination as to which of these different pathways through theinterconnected network a packet should travel is determined by the 2Lnetwork controller. In some embodiments, the 2L network controllerperforms optimization processing, in order to generate input tables thatspecify the optimized pathways. The virtualization application in the 2Lnetwork controller can then use these optimized pathways to generate theappropriate tunneling lookups to send packets along the correct paththrough the network.

FIG. 46 conceptually illustrates such optimization processing in ahigher-level network controller of some embodiments. Specifically, FIG.46 conceptually illustrates a pathway optimizer 4600 that receives (i)user-defined policies 4605 and (ii) a traffic matrix 4610, and outputsoptimized paths for packets through the network. For instance, in atwo-level network, this optimization processing would be performed in asecond level network controller in order to determine pathways throughthe first level domains.

As shown, in some embodiments the pathway optimizer 4600 receivesuser-defined policies 4605 from a user interface 4615. In someembodiments, the user interface 4605 is the same user interface throughwhich input tables to the control application are generated. The userinterface may be a graphical user interface, a command line interface,or other mechanism for allowing the user to input pathway data. Throughthe user interface, the user may input various policies. For instance,the user settings might specify that a certain quality of service (QoS)is required for packets sent from a particular machine, from aparticular network segment, to a particular destination machine ornetwork segment, etc. User settings can also specify that packetsto/from a particular source/destination should always be sent along aparticular path (overriding the optimization processing) or shouldalways receive the best connection. The user can specify differentclasses of traffic as well (e.g., high priority, medium priority, lowpriority), which has the effect of treating the packets differently froma QoS perspective.

In addition, the pathway optimizer 4610 receives a traffic matrix 4610from the lower level controllers (collectively represented as box 4620).The traffic matrix 4610 contains statistics regarding network datawithin and between the various 1L domains. In various embodiments, thistraffic matrix contains various information regarding the number ofpackets being sent out and received at various ports of the managedswitching elements of the 1L networks, the travel time of the packetsalong various paths within and between domains, etc. In someembodiments, this data is collected by the various managed switchingelements and then retrieved by the 1L network controllers (e.g., byquerying the managed switching elements). The 1L network controllers mayperform some processing on the data first or pass the raw data directlyto the higher-level network controllers, for use in the pathwayoptimizer. Some embodiments collect the network statistics on a regularbasis (e.g., every hour, every four hours, etc.) in order to regularlyupdate the pathways through the network and keep the pathways optimized.

The pathway optimizer 4600 generates a set of optimized paths 4625,which may be used as an input table by the logical forwarding plane ofthe higher-level controller in order to determine optimized pathwaysthrough the domain. Various different optimization algorithms are usedby different embodiments, such as simulated annealing, etc. Theoptimizer uses the policies 4605 as constraints and determines the bestset of pathways given the traffic matrix 4610. The most importantpackets (highest QoS) are given the best pathways, while the optimizerattempts to additionally optimize the lowest QoS packets as well,without harming the high QoS packets. These optimized paths 4625 areoutput for use by the virtualization application for generating lookupentries for the lower-level logical forwarding plane.

C. Packet Processing

While the above section describes the generation of forwarding tableentries for the interconnecting managed switching elements in a networkthat interconnects network segments at different network sites, thefollowing section will describe the processing of packets by thosemanaged switching elements using the generated flows. The subsectionswill describe the processing of packets by managed switching elements inboth a full mesh interconnecting network with a single logical datapathset and a hierarchical logical network with multiple levels of logicaldatapath sets.

1. Single Logical Layer

FIG. 47 conceptually illustrates the path of a packet 4700 through twomanaged switching elements between its source in a first network segmentand its destination in a second network segment. The operation of themanaged interconnection switching elements shown in this figure will bedescribed in part by reference to FIGS. 48 and 50, which conceptuallyillustrate processes performed by some of the managed switching elementsin such an interconnection network in order to process and forwardpackets.

As shown, the packet 4700 originates from a source machine (in a networksegment) with a payload 4705, headers 4710, and segment information4715. The payload 4705 contains the actual data intended for thedestination machine, while the headers 4710 include information appendedby the source machine in order to enable the packet 4700 to reach thedestination machine. For instance, the headers 4710 might include thesource and destination machines' physical addresses (e.g., MACaddresses) and/or network addresses (e.g., IP addresses).

The segment information 4715 is a context tag in some embodiments thatindicates the local network segment of the source machine. This segmentinformation may be a VLAN tag, MPLS header, MAC-in-MAC header, etc. Insome embodiments, one or more of the network segments may be a managednetwork, such as those shown in Section I, and the segment info is alogical context tag for the network.

The packet 4700 is sent from the source machine through its localnetwork (in most cases, through one or more switching elements withinthe local network). The local network switching elements will havelearned that packets sent to the destination address in the packetheader 4710 should be forwarded to the local interconnection switchingelement 4720. In some embodiments, the segment info 4715 includes adestination address field, with an indicator for remote destinationsthat the local switching elements recognize, causing them to forward thepacket to the local interconnection switching element 4720.

As shown, the local interconnection switching element 4720 (e.g., anextender at the edge of the local segmented network, that faces theexternal interconnecting network) first removes the local context taginformation (i.e., the segment info 4715). In some embodiments, theinterconnection switching element 4720 uses the decoding information forthe local segmented network stored in its tables. The switching element4720 then executes the logical flow for the interconnection network, inwhich it adds logical egress information 4725 to the packet 4700.

FIG. 48 conceptually illustrates in greater detail a process 4800 ofsome embodiments for processing packets by the source network'sinterconnection switching element. As indicated above, in someembodiments this switching element is an extender managed by a networkcontroller cluster.

As shown, the process 4800 begins by receiving (at 4805) a packet with adestination external to the local segmented network at the physicalingress port facing the local segmented network. A switching element,whether managed or unmanaged, has several physical ports through whichpackets may enter or exit. In general, each port can serve as both aningress port (for packets entering the switching element) and an egressport (for packets exiting the switching element), although in someembodiments certain ports may be reserved for either ingress or egressspecifically. In general, the extender or other interconnectionswitching element will have a single port that connects to the localsegmented network, though the switching element could have several suchports in some embodiments.

The process then removes (at 4810) the local network segment tagging. Asstated, the network segment context tag may be a VLAN tag, MPLS label,MAC-in-MAC header, etc., depending on the segmenting technique used bythe local network. The switching element uses its decoding informationto identify the local context tag portion of the packet in someembodiments. That is, the decoding information specifies which bits of apacket will have the local context tag, and what each bit within thecontext tag means. As such, the switching element will not only removethe context tag, but can store any relevant information in its registers(e.g., the specific VLAN on which the source machine resides).

Next, the process determines (at 4815) the logical ingress port of thepacket based on the physical ingress port and the segment tag. In someembodiments, this entails first identifying the logical datapath set towhich the packet belongs. For instance, in the example of FIGS. 39 and40, packets from either of the two VLANs 3520 and 3525 could arrive onthe same physical port of the managed switching element 3605 thatconnects to the network 3505. As such, the physical port alone cannot beused to determine the logical datapath set and ingress port in someembodiments. Instead, the switching element uses the network segmentcontext tag that was removed at operation 4810 to identify the logicaldatapath set and ingress port. In some embodiments, the switchingelement may also, or alternatively, use the source machine address toidentify the logical datapath set and ingress port.

Next, the process 4800 identifies (at 4820) a destination network forthe packet. In some embodiments, the managed switching element uses thedestination address (e.g., a MAC address, IP address, etc.) stored inthe packet header and matches this destination address to one of thenetwork segments connected by the logical datapath set identified atoperation 4815. As shown in FIG. 40 above, in some embodiments themanaged switching element stores a list of all known addresses for eachof the network segments connected by each of the logical datapath setsthat the switching element implements. In some embodiments, theswitching element uses a traditional flooding-based learning algorithmto handle packets for which it does not recognize the destination inorder to determine on which network segment a particular machine islocated.

The process next determines (at 4825) a logical egress port for theidentified destination network on the logical datapath set. In someembodiments, the identification of the destination network and thesubsequent identification of the logical egress port are performed as asingle operation by the managed switching element. In addition to makinga forwarding decision (i.e., mapping to a logical egress port), someembodiments also perform other forwarding table operations within thelogical processing. For instance, some embodiments perform ACL lookupsthat may contain instructions to drop the packet, enqueue the packet(e.g., to enforce quality of service controls), allow a packet through,etc. In some embodiments, operations 4820 and 4825 are performed as asingle operation (i.e., as a single lookup). That is, the managedswitching element executes a forwarding table lookup entry that simplymatches the destination network of the packet to a logical egress port,without having a separate operation for identifying the destinationnetwork.

After determining the logical egress port, the process encapsulates (at4830) the packet with this logical egress port information. That is, themanaged switching element prepends information to the packet (e.g., alogical context) that includes the egress port information. An exampleof such a logical context for OSI Layer 2 processing is described indetail in U.S. application Ser. No. 13/177,535, incorporated byreference above. This logical context is a 64-bit tag that includes a32-bit virtual routing function field (for representing the logicaldatapath set to which the packet belongs), a 16-bit logical inport field(i.e., the ingress port of the datapath that corresponds to the localnetwork segment), and a 16-bit logical outport field (i.e., theidentified logical egress port that corresponds to the destinationremote network segment).

At this point, the forwarding decisions for the packet are complete, andthe process 4800 transmits (at 4835) the encapsulated packet towards thephysical location of the logical egress port, and ends. This location,in some embodiments, is an interconnection switching element (e.g.,extender) at the edge of the network containing the destination networksegment. In some embodiments, this transmission actually involvesseveral operations. First, the logical egress port is mapped to aphysical address (e.g., the address of a port on the interconnectionswitching element). Next, this physical address is mapped to a physicalport of the managed interconnection switching element performing theoperations so that the packet can be transmitted to the next hop. Whilethe remote interconnection switching element is the ultimate destination(as far as the logical network is concerned), there will generally beseveral physical switching elements in between the sourceinterconnection switching element and the remote interconnectionswitching element. In some embodiments, a tunnel is defined between thetwo switching elements, and the packet is encapsulated with the tunnelinformation to be sent over the intervening network.

FIG. 49 conceptually illustrates an example of some of the forwardingtable operations performed by a source interconnection switching element4900 (i.e., the managed interconnection switching element that connectsto a network containing the source of a packet). Specifically, FIG. 49illustrates forwarding table entries 4905 for the source interconnectionswitching element 4900.

In conjunction with the forwarding table entries, FIG. 49 conceptuallyillustrates the processing pipeline 4950 performed by the sourceinterconnection switching element 4900 of some embodiments. As shown bythe numbers 1-4, when the interconnection switching element 4900receives a packet, it uses numerous forwarding table entries to processthe packet. In some embodiments, the physical and logical tables(including any ACL tables) are implemented as a single table within themanaged switching element (e.g., using a dispatch port that returns apacket processed by a first entry to the forwarding table for processingby a second entry).

The VMs in this example refer to those shown in FIG. 39. As shown, VM 1sends a packet 4910 that arrives at the source interconnection switchingelement 4900 (corresponding to the interconnection switching element3605). This packet, in most cases, will not have been sent directly fromthe VM to the interconnection switching element, but will usually havetraveled through at least one switching element within the segmentednetwork between the VM and the interconnection switching element.

The managed switching element 4900 receives the packet 4910 through aninterface of the switching element, and begins processing the packetusing the forwarding tables 4905. The first stage in the processingpipeline 4950 is an ingress context mapping stage 4955 that maps aphysical ingress port (i.e., the interface through which the packet wasreceived from VM 1) and packet information (e.g., the VLAN tags) to alogical ingress port (i.e., a port of one of the logical switchingelements implemented by the physical switching element that correspondsto the source VLAN).

As shown by the encircled 1, the interconnection switching elementidentifies a record 1 in the forwarding table that implements theingress context mapping. Specifically, this record identifies thenetwork segment from which the packet was sent (VLAN 1) and matches thisnetwork segment to a logical port of a particular logical datapath set(Port 1 of LDPS A). In this case, the forwarding tables 4905 includeadditional records for matching different source VMs on the same anddifferent VLANs to logical ingress ports—different source VMs on thesame VLAN will be matched to the same port, while source VMs ondifferent VLANs are matched to ports of different logical switchingelements. Some embodiments only store one record for each networksegment for ingress port matching. That is, the forwarding tables do notcare from which VM the packet originated, only using the VLAN (or othernetwork segment) information. In this case, the record 1 specifies thatthe managed interconnection switching element 4900 store the logicalcontext of the packet in a field of the packet's header, which indicatesthe logical inport of a particular logical datapath set for the packet.In some embodiments, the record also specifies to send the packet to thedispatch port, for additional processing by the forwarding tables 4905

The second stage in the processing pipeline 4950 is the logicalforwarding lookups 4960. As in the federated network examples shownabove, the forwarding lookups 4960 are illustrated here as a singleforwarding table record, but may actually involve several differentrecords performing several different functions. For instance, the tables4905 do not illustrate any ACL tables, which might be present to enforcesecurity policies for packets sent by VM 1 or from VLAN 1, or sent toparticular VMs.

As shown by the encircled 2, the interconnection switching element 4900identifies a record 2 in the forwarding tables 4905 that implements thelogical forwarding decision. Specifically, this record identifies thepacket destination (specified in the packet header) as a particular VM13 and sets the logical egress of the packet to be a particular port(Port 2) of the already-identified logical datapath set. As shown, insome embodiments the forwarding table includes records for each of thepossible destination addresses (i.e., each of the machines on thedifferent network segments connected by the interconnection network).Thus, the forwarding table 4905 includes records for setting the logicalegress to the same Port 2 of the same logical datapath set when thedestination is VM 14, and a different Port 3 of the same logicaldatapath set when the packet destination is VM 20. In some embodiments,the record instead specifies that when a network segment (e.g., MPLSLabel 2) is matched based on a list of all of the VMs on that networksegment, the record maps the network segment to a particular port. Therecord 2 specifies that the managed switching element 4900 store thelogical egress port in the packet headers (i.e., encapsulate the packetwith the logical egress context), as well as send the packet to itsdispatch port.

Based on the logical egress port specified at the second stage of theprocessing pipeline, the managed switching element performs egresscontext mapping 4965 that maps the logical egress port to a physicalegress port for the packet within the interconnecting network. For apacket traveling from one network segment at a first site to a differentnetwork segment at a second site, this will be the physical port of adifferent interconnection switching element at the second site. As shownby the encircled 3, the source interconnection switching element 4900identifies a record 3 in the forwarding tables 4905 that implements theegress context mapping. Specifically, the record 3 matches the logicalegress as Port 2 of LDPS A, and sets the destination to be a particularextender 2 that faces the MPLS-segmented site containing Label 2. Insome embodiments, the destination is set as the MAC address of aparticular port (i.e., the port facing the interconnecting network) ofthe extender 2. In some embodiments, this involves encapsulating thepacket in a tunnel between the managed interconnection switching element4900 and the extender 2 specified as the physical egress port. In someembodiments, the record 3 also specifies to send the packet to itsdispatch port for further processing.

Finally, the managed interconnection switching element 4900 performs thephysical mapping stage 4970 that specifies a physical port of themanaged switching element through which to send the (now-modified)packet 4910 in order to reach the physical egress port identified by theegress context mapping. As shown by the encircled 4, the interconnectionswitching element 4900 identifies a record 4 in the forwarding tables4905 that implements this physical mapping. Specifically, the record 4matches the destination (extender 2) and maps the packet to a port 2 ofthe interconnection switching element 4900 (referred to in the table as“extender 1”). This port is different from the dispatch port, andtherefore the packet is now sent out this port towards the destinationinterconnection switching element, for eventual delivery to VM 13.

Returning to FIG. 47, the packet is sent from the local interconnectionswitching element 4720 through the interconnecting network to the remoteinterconnection switching element 4730 (i.e., through a tunnel definedbetween ports of the two switching elements). The remote interconnectionswitching element executes the logical flow for the interconnectionnetwork, which involves removing the logical egress encapsulation (asthe packet has reached its destination, as far as the logical network isconcerned). The interconnection switching element 4730 also adds newsegment context information 4735, so that the packet can be processed byswitching elements on the receiving segmented network and delivered toits destination machine.

FIG. 50 conceptually illustrates in greater detail a process of someembodiments for processing packets by the destination network'sinterconnection switching element. As indicated above, in someembodiments this switching element is an extender managed by a networkcontroller cluster (i.e., the same network controller cluster thatmanages an extender at the source site).

As shown, the process 5000 begins by receiving (at 5005) an encapsulatedpacket at a physical port that faces an external interconnecting networkwith a destination in a segmented network local to the receivingswitching element. In some embodiments, however, the destination cannotyet be determined, because the actual end machine destination is hiddenwithin the encapsulation. In many cases, the physical switching elementwill have several ports that face away from the local segmented network,but the tunnel between the sending interconnection switching element andthe receiving interconnection switching element is defined betweenparticular ports on each switching element.

The process then removes (at 5010) the encapsulation on the packet. Insome embodiments, the encapsulation includes a tunneling protocol usedto send the packet to the particular receiving interconnection switchingelement, as well as the logical egress context of the packet. Theswitching element recognizes its ingress port as the end of the tunnel,and therefore removes the tunneling encapsulation, and additionallyrecognizes itself as the logical egress port for the packet, removingthe logical egress information. In some embodiments, the switchingelement stores the logical context information in registers, in case theinformation is needed for further processing.

Next, the process 5000 identifies (at 5015) a destination in thesegmented network local to the interconnection switching element. Insome embodiments, the switching element uses the removed logical contextto map to a particular segmented network. The logical egress port,information about which was removed at operation 5010, corresponds to aparticular network segment within the local site network. Someembodiments, on the other hand, use the address of the destinationmachine (e.g., the MAC address) to identify the local network segment towhich the packet should be sent.

The process then adds (at 5020) the context tags for the local segmentednetwork. These tags may be a VLAN tag, MPLS label, MAC-in-MAC header,etc., depending on the segmenting technique used by the local network.The switching element uses its decoding information, in someembodiments, to determine which bits of the packet should be used forthe different portions of the context tag. These portions may include anetwork identifier (e.g., a VLAN ID), a destination on the VLAN, etc.

Finally, the process transmits (at 5025) the packet onto the localsegmented network towards the packet's physical destination, then ends.In most cases, the packet will not be sent directly from the managedinterconnection switching element to the end machine (physical machineor virtual machine) that is the packet's destination. Instead, therewill likely be one or more intervening switching elements on the localnetwork that process the packet according to its network segment contexttags.

FIG. 51 conceptually illustrates an example of some of the forwardingtable operations performed by a destination interconnection switchingelement 5100 (i.e., the managed interconnection switching element thatconnects to a network containing the destination of a packet).Specifically, FIG. 51 illustrates forwarding table entries 5105 for thedestination interconnection switching element 5100.

In conjunction with the forwarding table entries, FIG. 51 conceptuallyillustrates the processing pipeline 5150 performed by the sourceinterconnection switching element 5100 of some embodiments. Thisprocessing pipeline 5150 contains the same four stages as the pipeline4950 shown in FIG. 49 for the source interconnection switch: ingresscontext mapping, then logical forwarding, then egress context mapping,and physical forwarding. In some embodiments, each managed switchingelement in a single-level logical network performs the same processingpipeline, though some of the stages may not actually involve theperformance of any operations (e.g., at managed switching elements inthe middle of a network that simply pass a packet onwards to the nextmanaged switching element).

Thus, the first forwarding table record identified by theinterconnection switching element 5150 implements the ingress contextmapping stage. This record, shown by the encircled 1, identifies thatthe logical context has already been set to a particular logicaldatapath set, and performs no additional operation, while simply sendingthe packet to the dispatch port (and, in some embodiments, storing thisinformation to a register). Next, the switching element identifies aforwarding table record that identifies that (i) the packet alreadyincludes its logical egress information and (ii) that information can beremoved because the packet has now reached its logical egress. At theegress context mapping stage, the switching element identifies thatlogical egress corresponds to a particular network segment on its localnetwork and that the destination is a particular machine (VM 13) on thatnetwork segment. Finally, at the physical mapping stage, the switchingelement identifies a particular physical port to which it sends thepacket, in order for the packet to reach its destination on thesegmented network.

2. Several Logical Layers

In the above scenario, only two managed switching elements are generallyinvolved for a particular packet: the interconnection switching elementat the source network and the interconnection switching element at thedestination network, with a tunnel defined between the two. On the otherhand, in the multi-level interconnection scenario, with at least twolayers of logical datapath sets, in some cases a packet may travelthrough numerous such interconnection switching elements, enablingvarious traffic forwarding decisions. In this scenario, the second levelcontroller can make decisions about which of several paths a packetshould take to travel from a source network in a first 1L domain to adestination network in a second 1L domain (e.g., through one or theother of two possible intervening 1L domains). The second levelcontroller pushes these decisions down to the first level controllers,which implement the decisions within their first level flows sent to theinterconnection switching elements within their respective domain.

In Subsection A of this Section, a simplistic example was illustratedfor the case of interconnecting network segments via multiple levels oflogical datapath set. FIG. 52 conceptually illustrates a more complexnetwork 5200, with four separate 1L domains of three interconnectednetworks each. For simplicity, the segmented networks located behind thetwelve interconnection switching elements are not shown in this figure.Instead, only a first VM 5205 and a second VM 5210 are illustrated, asthese will be used to describe an example packet flow.

The hierarchical network 5200 includes a first 1L domain 5215 (“West”)with three interconnection switching elements 5216-5218, a second 1Ldomain 5220 (“North”) with three interconnection switching elements5221-5223, a third 1L domain 5225 (“South”) with three interconnectionswitching elements 5226-5228, and a fourth 1L domain 5230 (“East”) withthree interconnection switching elements 5231-5233. Each of theseinterconnection switching elements is located at the edge of a segmentednetwork (e.g., a network segmented using VLANs, MPLS Labeling,MAC-in-MAC, etc.). In some embodiments, the segmented networks may alsouse the logical network virtualization as described in Section I. Eachof the four 1L domains includes a 1L network controller cluster (notshown) that generates flows and pushes the flows to the interconnectionswitching elements within the 1L domain. These four 1L networkcontroller clusters are all connected to a 2L network controllercluster, that generates 2L flows and pushes the flows to the 1Lcontroller clusters for incorporation within their flows. In addition,the 2L network controller cluster may make decisions about the preferredpath of several different paths through the 1L domains for packets fromdifferent source networks or machines and to different destinationnetworks or machines.

FIG. 53 illustrates an example of a packet 5300 traveling through thenetwork 5200 from VM 1 5205 to VM 2 5210. In this case, the packettravels through six of the interconnection switching elements, labeledin this figure as extenders. As shown, the packet originates at VM 15205 with its payload and headers 5305 (e.g., source and destination MACaddress, source and destination IP address, etc.), as well as a VLANcontext tag 5310 identifying the packet as belonging to VLAN 1 (the VLANto which the source VM 1 belongs). In most cases, the packet will havetraveled through at least one switching element within the segmentednetwork that contains VLAN 1 before arriving at the first extender 5216.

Upon receiving the packet 5300, the extender 5216 first removes the VLANcontext tag 5305, and identifies the 1L and then 2L logical ingressports (performing ingress context mapping). The extender performs the 2Llogical forwarding decisions, including any ACL decisions. The policydecisions implemented by the ACL flow entries may be generated by the 2Lcontroller and then pushed down to the 1L controller and then to themanaged switching elements, as described above. In addition, the 2Llogical forwarding identifies an 2L egress port, which corresponds to aparticular port on extender 5232 to which the destination VM 2 5210attaches. As in the single-layer scenario from the previous subsection,the extender 5216 stores a list of the ports to which differentdestination addresses (e.g., MAC addresses) correspond.

After encapsulating the packet with 2L egress information 5315, theextender 5216 executes the remaining 1L processing pipeline. Thisincludes egress context mapping of the 2L egress port to a particular 1Legress port. In this case, the packet can arrive at its destination (inthe East 1L domain) via either the North domain or the South domain. Insome embodiments, this decision is determined by the 2L controller,which determines which path should be taken through the 1L datapaths forpackets from different source networks or machines and to differentdestination networks or machines. The 2L controller of some embodimentsmight specify that a packet from a particular source (either machine ornetwork segment) to a particular destination (either machine or networksegment) should travel along a specific set of 1L datapaths. For theoriginating 1L datapath, this will be implemented by specifying anegress port of the 1L datapath that sends the packet to the “next hop”1L datapath. Upon arriving at that next 1L datapath, the 1L egress portwill be determined such that the packet is again sent to the appropriatenext 1L datapath. Within the 1L datapath, forwarding decisions betweenphysical interconnection switching elements are determined by the 1Lcontroller cluster.

In order to generate the forwarding lookups, the 2L controller clusterof some embodiments combines user input specifying policies for trafficas well as network statistics (i.e., a traffic matrix) received from the1L controllers. In some embodiments, the lower-level controllersautomatically collect information regarding their respective regions ofthe interconnecting network (i.e., from the switching elementsimplementing their lower-level logical datapath set), and pass thisinformation upward to the second level network controller. In someembodiments, these statistics relate to the number of packets being sentout and received at various ports of the managed switching elements, thetravel time of the packets along different paths, etc.

In different embodiments, the user policies might specify differentclasses of traffic (e.g., high, medium, and low priority packets),guarantee a particular QoS for specific packets while making noguarantees about other packets, direct specific packets to follow aparticular path, etc. The packets might be classified based on sourcemachine, source network segment, source 1L datapath, destinationmachine, destination network segment, destination 1L datapath, or acombination thereof, in different embodiments. For instance, a usermight specify packets from a VLAN 5 at Network A to a VLAN 6 at NetworkB to be high priority packets, while packets from VLAN 5 at Network A toa VLAN 3 at Network C are low priority packets. As another example, theuser might specify a particular guaranteed QoS for all packets from afirst 1L datapath to a second 1L datapath, or from a particular sourcemachine (irrespective of destination). One of ordinary skill in the artwill recognize that many different combinations are possible fordetermining policy.

The 2L controller (or associated processing) performs an optimizationroutine (e.g., a constrained optimization algorithm) to generate thepaths through the 1L datapaths. The optimization routine attempts toidentify an optimal set of paths for all different possible packets thatbest carries out the user-set policies based on the most recent networkstatistics. For instance, if a particular connection to a particular 1Ldatapath does not appear reliable, then the optimization routine willrequire the higher-priority traffic to follow a different path that doesnot travel over the unreliable connection, so long as a better option isavailable. For low priority packets, the optimization routine willgenerally choose the best option that will not interfere with the higherpriority packets.

In the case illustrated in FIG. 53, the optimization performed by the 2Lcontroller has specified that the particular packet should be sent viathe South 1L domain 5225, rather than the North 1L domain 5220.Accordingly, the extender 5216 adds the West 1L egress information 5320,which specifies the logical egress port connecting to the South 1Ldomain 5225. The extender 5216 maps this logical egress port to a portof the extender 5218 and transmits the packet out of its port through atunnel to the extender 5218.

The packet travels through the West 1L network and arrives at theextender 5218, the location of the West 1L egress port towards the South1L domain. This extender 5218 removes the West 1L egress encapsulation5320, and executes interconnection instructions contained within itsforwarding tables. The interconnection instructions specify the South 1Lingress port (corresponding to a port on extender 5226), and theswitching element 5218 encapsulates the packet with this ingressinformation 5325 and sends the packet through the interconnectingnetwork (e.g., through several non-managed switching elements) to theextender 5226.

The receiving interconnection switching element 5226 in the South 1Ldomain (and, in a more complex example, any additional intervening 1Ldomains between the source and destination 1L domains) does not modifythe 2L encapsulation. Instead, the extender 5226 receives the packet,removes the South 1L ingress information, and maps the 2L egress port toits own 1L egress port that corresponds to a next 1L datapath. As in theextender 5216, this 1L mapping decision is governed by the lookupentries provided by the 2L controller. The extender 5226 encapsulatesthe packet with its South 1L egress information 5330 for transmissionthrough the South 1L domain to the extender 5227.

The operations performed by the extender 5227 are parallel to thoseperformed by the extender 5218 at the edge of the West 1L datapath. Thisinterconnection switching element removes the South 1L egressinformation (as the packet 5300 will be exiting the South 1L domain) andadds East 1L ingress information 5335. The extender 5227 then sends thepacket through the interconnecting network to the ingress port of theEast 1L domain at extender 5231.

At the East 1L domain, the extender 5231 first removes the 1L ingressinformation. Next, the extender runs the 2L pipeline, which maps the 2Legress port to the East 1L port that corresponds to the extender 5232.As the 2L egress port now maps to a port facing a segmented network,rather than a different 1L domain, the extender 5231 removes the 2Lencapsulation as well. The extender adds the 1L egress information 5340,and transmits the packet through the East 1L network to the finalextender 5232.

This extender 5232 includes a port to which a segmented network,including the segment VLAN 2, connects. The destination machine VM 25210 belongs to this VLAN 2. Upon receiving the packet 5300, theextender 5232 removes the 1L encapsulation, and uses the information itstores for its network segments (including VLAN 2) to add a context tag5345 for VLAN 2 to the packet. The extender then forward the packet ontothe segmented network for its eventual arrival at the destination VM 25210.

As mentioned, the above example of a two level interconnecting networkis a somewhat simplistic example, with only two possible paths betweenthe East and West 1L domains. As with the hierarchical networksdescribed in Section II, in some embodiments more than two levels may beused to divide a network into separate datapaths. For instance, someembodiments might use 1L datapaths to connect networks within a buildingor a city block, 2L datapaths to connect 1L datapaths within a city, 3Ldatapaths to connect 2L datapaths within a region (e.g., a state), 4Ldatapaths to connect 3L datapaths within a country, etc. One of ordinaryskill in the art will understand that different geographic regions, orother mechanisms for grouping networks, may be used.

Within such a hierarchical network, the first hop processing at eachlogical datapath set of a given level will use a set of packetforwarding instructions from the higher level controller to identify thenext logical datapath set at the given level. That is, just as the setof forwarding instructions from the 2L controller in the examplegoverned the choice of a next 1L datapath each time a packet entered an1L datapath, an 3L controller would generate a set of packet forwardinginstructions to govern the choice of a next 2L datapath for packetstraveling across 2L datapaths. Thus, at the first hop in a three-levelinterconnection network, the 3L processing would identify an 3L egressport (for this description, assumed to be located in a different 2Ldatapath). Based on a set of forwarding instructions from the 3Lcontroller, this 3L egress port maps to an 2L egress port connecting toa particular next 2L datapath. The 2L network controller would also havea generated set of forwarding instructions that describes how to forwardthe packet internal to its 2L domain in order to get the packet from thecurrent 1L domain to the 1L domain through which the packet exits thedatapath.

D. Learning in Interconnected Network

The above examples describe situations in which the variousinterconnection switching elements already knew the locations of thedestination machines for the packets. However, as machines are added toa network or moved from one network segment to another, in some casesthe interconnection switching elements may not have the requiredinformation to make the forwarding decision to a particular logicalegress port. In some embodiments, the interconnection switching elementsuse standard learning mechanisms, and flood the network ofinterconnection switching elements in order to identify behind which ofthe other switching elements a particular destination MAC address islocated.

In some embodiments, however, alternatives to the standardlearning-by-flooding may be used. A centralized solution of someembodiments uses the network controller clusters to accumulate anddistribute data. In such an approach, the interconnection switchingelements (e.g., extenders) at each segmented network report theaddresses (e.g., MAC addresses) of the machines seen at their sitenetworks to the virtualization application of the network controller.The virtualization application reports these addresses (via its logicaldatapath set abstraction) to the centralized control plane of thenetwork controller. The network controller can then compute thenecessary updates to the forwarding table (i.e., populating the lists ofMAC addresses to associate with different logical ports shown in FIG.41). In effect, in the centralized approach, the interconnectionswitching elements all report the MAC addresses on their differentnetwork segments to the network controller, and the network controllerthen distributes this information to the other extenders.

Other embodiments use a decentralized approach to learning that does notrequire flooding. In this approach, the interconnection switchingelements communicate directly, rather than through the central networkcontroller. The interconnection switching elements distribute, to theother interconnection switching elements, lists of the addresses theyhave seen at their respective network segments. The addresses may bedisseminated whenever there is a change, in some embodiments (e.g., anew machine with a new address appears in a segmented network).

An additional decentralized approach of some embodiments builds on adistributed lookup service that provides an ability to execute a lookupfor an address to site location binding. That is, a lookup serviceexists (in a distributed fashion) that stores the lists of addresses fordifferent network segments. The interconnection switching elements cansend an unresolved address to the lookup service and receive anidentification of the network segment at which that address is located.

In some embodiments, the same considerations apply to replication of anaddress resolution protocol (ARP) database across the sites. Scalingthis replication, in some embodiments, requires replacing the standardflooding-based mechanism with one of the centralized or decentralizedapproaches described above.

IV. Computer System

Many of the above-described features and applications are implemented assoftware processes that are specified as a set of instructions recordedon a computer readable storage medium (also referred to as computerreadable medium). When these instructions are executed by one or moreprocessing unit(s) (e.g., one or more processors, cores of processors,or other processing units), they cause the processing unit(s) to performthe actions indicated in the instructions. Examples of computer readablemedia include, but are not limited to, CD-ROMs, flash drives, RAM chips,hard drives, EPROMs, etc. The computer readable media does not includecarrier waves and electronic signals passing wirelessly or over wiredconnections.

In this specification, the term “software” is meant to include firmwareresiding in read-only memory or applications stored in magnetic storagewhich can be read into memory for processing by a processor. Also, insome embodiments, multiple software inventions can be implemented assub-parts of a larger program while remaining distinct softwareinventions. In some embodiments, multiple software inventions can alsobe implemented as separate programs. Finally, any combination ofseparate programs that together implement a software invention describedhere is within the scope of the invention. In some embodiments, thesoftware programs, when installed to operate on one or more electronicsystems, define one or more specific machine implementations thatexecute and perform the operations of the software programs.

FIG. 54 conceptually illustrates a computer system 5400 with which someembodiments of the invention are implemented. The electronic system 5400may be a computer, server, dedicated switch, phone, or any other sort ofelectronic device. Such an electronic system includes various types ofcomputer readable media and interfaces for various other types ofcomputer readable media. Electronic system 5400 includes a bus 5405,processing unit(s) 5410, a system memory 5425, a read-only memory 5430,a permanent storage device 5435, input devices 5440, and output devices5445.

The bus 5405 collectively represents all system, peripheral, and chipsetbuses that communicatively connect the numerous internal devices of theelectronic system 5400. For instance, the bus 5405 communicativelyconnects the processing unit(s) 5410 with the read-only memory 5430, thesystem memory 5425, and the permanent storage device 5435.

From these various memory units, the processing unit(s) 5410 retrieveinstructions to execute and data to process in order to execute theprocesses of the invention. The processing unit(s) may be a singleprocessor or a multi-core processor in different embodiments.

The read-only-memory (ROM) 5430 stores static data and instructions thatare needed by the processing unit(s) 5410 and other modules of theelectronic system. The permanent storage device 5435, on the other hand,is a read-and-write memory device. This device is a non-volatile memoryunit that stores instructions and data even when the electronic system5400 is off. Some embodiments of the invention use a mass-storage device(such as a magnetic or optical disk and its corresponding disk drive) asthe permanent storage device 5435.

Other embodiments use a removable storage device (such as a floppy disk,flash drive, or ZIP® disk, and its corresponding disk drive) as thepermanent storage device. Like the permanent storage device 5435, thesystem memory 5425 is a read-and-write memory device. However, unlikestorage device 5435, the system memory is a volatile read-and-writememory, such a random access memory. The system memory stores some ofthe instructions and data that the processor needs at runtime. In someembodiments, the invention's processes are stored in the system memory5425, the permanent storage device 5435, and/or the read-only memory5430. From these various memory units, the processing unit(s) 5410retrieve instructions to execute and data to process in order to executethe processes of some embodiments.

The bus 5405 also connects to the input and output devices 5440 and5445. The input devices enable the user to communicate information andselect commands to the electronic system. The input devices 5440 includealphanumeric keyboards and pointing devices (also called “cursor controldevices”). The output devices 5445 display images generated by theelectronic system. The output devices include printers and displaydevices, such as cathode ray tubes (CRT) or liquid crystal displays(LCD). Some embodiments include devices such as a touchscreen thatfunction as both input and output devices.

Finally, as shown in FIG. 54, bus 5405 also couples electronic system5400 to a network 5465 through a network adapter (not shown). In thismanner, the computer can be a part of a network of computers (such as alocal area network (“LAN”), a wide area network (“WAN”), or an Intranet,or a network of networks, such as the Internet. Any or all components ofelectronic system 5400 may be used in conjunction with the invention.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in amachine-readable or computer-readable medium (alternatively referred toas computer-readable storage media, machine-readable media, ormachine-readable storage media). Some examples of such computer-readablemedia include RAM, ROM, read-only compact discs (CD-ROM), recordablecompact discs (CD-R), rewritable compact discs (CD-RW), read-onlydigital versatile discs (e.g., DVD-ROM, dual-layer DVD-ROM), a varietyof recordable/rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.),flash memory (e.g., SD cards, mini-SD cards, micro-SD cards, etc.),magnetic and/or solid state hard drives, read-only and recordableBlu-Ray® discs, ultra density optical discs, any other optical ormagnetic media, and floppy disks. The computer-readable media may storea computer program that is executable by at least one processing unitand includes sets of instructions for performing various operations.Examples of computer programs or computer code include machine code,such as is produced by a compiler, and files including higher-level codethat are executed by a computer, an electronic component, or amicroprocessor using an interpreter.

While the above discussion primarily refers to microprocessor ormulti-core processors that execute software, some embodiments areperformed by one or more integrated circuits, such as applicationspecific integrated circuits (ASICs) or field programmable gate arrays(FPGAs). In some embodiments, such integrated circuits executeinstructions that are stored on the circuit itself.

As used in this specification and any claims of this application, theterms “computer”, “server”, “processor”, and “memory” all refer toelectronic or other technological devices. These terms exclude people orgroups of people. For the purposes of the specification, the termsdisplay or displaying means displaying on an electronic device. As usedin this specification and any claims of this application, the terms“computer readable medium” and “computer readable media” are entirelyrestricted to tangible, physical objects that store information in aform that is readable by a computer. These terms exclude any wirelesssignals, wired download signals, and any other ephemeral signals.

While the invention has been described with reference to numerousspecific details, one of ordinary skill in the art will recognize thatthe invention can be embodied in other specific forms without departingfrom the spirit of the invention. In addition, a number of the figures(including FIGS. 16, 17, 21, 22, 23, 28, 29, 44, 48, and 50)conceptually illustrate processes. The specific operations of theseprocesses may not be performed in the exact order shown and described.The specific operations may not be performed in one continuous series ofoperations, and different specific operations may be performed indifferent embodiments. Furthermore, the process could be implementedusing several sub-processes, or as part of a larger macro process.

We claim:
 1. A non-transitory machine readable medium storing a programwhich when executed by at least one processing unit implements a networkcontroller for managing a plurality of interconnection switchingelements in a network, the program comprising sets of instructions for:receiving a definition of a logical switching element that connects aset of network segments located at different segmented networks, thedefinition binding each logical port of the logical switching element toa different network segment, wherein each different network segmentcomprises a separate plurality of machines having unique destinationaddresses; and configuring a set of interconnection switching elementslocated at each of the different segmented networks to implement thelogical switching element by distributing forwarding instructions to theinterconnection switching elements, each of the interconnectionswitching elements connecting a different one of the network segments toa same interconnecting network to which each of the interconnectionswitching elements connects, wherein the forwarding instructions specifyto forward packets having any of the destination addresses of themachines for a particular network segment through the interconnectingnetwork to the interconnection switching element that connects theparticular network segment to the interconnecting network.
 2. Thenon-transitory machine readable medium of claim 1, wherein the set ofinstructions for receiving the definition of the logical switchingelement comprises a set of instructions for receiving the plurality ofunique destination addresses attached to each network segment in the setof network segments.
 3. The non-transitory machine readable medium ofclaim 1, wherein each of the segmented networks is a segmented L2network and the common interconnecting network is a L3 network.
 4. Thenon-transitory machine readable medium of claim 1, wherein a firstinterconnection switching element that connects a first one of thesegmented networks to the interconnecting network removes segmentationinformation of a packet received from the first segmented network toidentify a first network segment of a source machine of the packet, mapsthe identified network segment to the logical switching element, andencapsulates the packet with an identifier for a logical egress port ofthe logical switching element that corresponds to a second networksegment of a second segmented network.
 5. The non-transitory machinereadable medium of claim 4, wherein the definition of the logicalswitching element comprises a description of network segmentationtechniques for each of the segmented networks.
 6. The non-transitorymachine readable medium of claim 4, wherein a second interconnectionswitching element that connects the second network segment to theinterconnecting network receives the encapsulated packet from the firstinterconnection switching element, removes the encapsulation from thepacket in order to identify that the packet destination is a machine onthe second segmented network, adds segmentation information for thesecond segmented network to the packet.
 7. The non-transitory machinereadable medium of claim 1, wherein the interconnection switchingelements are extenders.
 8. A network control system for interconnectinga plurality of separate networks, the system comprising: a plurality ofnetwork segments comprising at least (i) a first network segment of afirst network segmented using a first technique and (ii) a secondnetwork segment of a second network segmented using a second technique,each network segment of the plurality of network segments comprising aplurality of machines; a set of interconnection switching elements, eachinterconnection switching element having a first physical port thatinterfaces with a particular one of the network segments and a secondphysical port that interfaces with an interconnecting network to whichall of the interconnection switching elements connect to provideconnections between the plurality of network segments; and a networkcontroller for managing the set of interconnection switching elements bydefining a logical switching element comprising logical ports to whicheach of the network segments couple, wherein the logical switchingelement is for implementation by each of the interconnection switchingelements of the set of interconnection switching elements, with eachlogical port corresponding to one of the interconnection switchingelements, wherein the plurality of machines of each network segment aremapped to the logical port for the network segment by theinterconnection switching elements.
 9. The network control system ofclaim 8, wherein the first network segment is segmented using virtuallocal area network (VLAN) tagging and the second network segment issegmented using multi-protocol labeling system (MPLS) labeling.
 10. Thenetwork control system of claim 8, wherein the first network segment issegmented using a first VLAN technique and the second network segment issegmented using a second VLAN technique.
 11. The network control systemof claim 8, wherein each interconnection switching element that has afirst physical port interfacing with a particular network segment storesinstructions for decoding and removing network segmentation informationfrom packets received from the particular network segment.
 12. Thenetwork control system of claim 8, wherein each interconnectionswitching element that has a first physical port interfacing with aparticular network segment stores instructions for adding networksegmentation information to packets received from other interconnectionswitching elements and directed towards machines of the particularnetwork segment.
 13. The network control system of claim 8, wherein thefirst network segment is located at a first datacenter and the secondnetwork segment is located at a second, different datacenter.
 14. Thenetwork control system of claim 8, wherein the logical switching elementcomprises a first logical port coupled to the first network segment anda second logical port coupled to the second network segment.
 15. Thenetwork control system of claim 14, wherein the plurality of networksegments comprises a third network segment of a third network segmentedusing a third technique, wherein the logical switching element comprisesa third logical port coupled to the third network segment.
 16. For anetwork controller, a method for managing a plurality of interconnectionswitching elements in a network, the method comprising: receiving adefinition of a logical switching element that connects a set of networksegments located at different segmented networks, the definition bindingeach logical port of the logical switching element to a differentnetwork segment, wherein each different network segment comprises aseparate plurality of machines having unique destination addresses; andconfiguring a set of interconnection switching elements located at eachof the different segmented networks to implement the logical switchingelement by distributing forwarding instructions to the interconnectionswitching elements, each of the interconnection switching elementsconnecting a different one of the network segments to a sameinterconnecting network to which each of the interconnection switchingelements connects, wherein the forwarding instructions specify toforward packets having any of the destination addresses of the machinesfor a particular network segment through the interconnecting network tothe interconnection switching element that connects the particularnetwork segment to the interconnecting network.
 17. The method of claim16, wherein receiving the definition of the logical switching elementcomprises receiving the plurality of unique destination addressesattached to each network segment in the set of network segments.
 18. Themethod of claim 16, wherein the definition of the logical switchingelement comprises logical control plane data, wherein configuring theset of interconnection switching elements comprises: converting thelogical control plane data to logical forwarding plane data; convertingthe logical forwarding plane data to physical control plane data; andpushing the physical control plane data to the interconnection switchingelements.
 19. The method of claim 16, wherein the definition of thelogical switching element comprises a description of networksegmentation techniques for each of the segmented networks.
 20. Themethod of claim 18, wherein the physical control plane data comprisesflow entries that enable the interconnection switching elements toimplement the logical switching element for packets received from andsent to machines in the different network segments.